Audio generation model training method and device, computer device, and storage medium

Abstract

The application relates to the technical field of machine learning, and discloses a training method and device of an audio generation model, a computer device and a storage medium. The method comprises the following steps: obtaining training data, wherein the training data comprises audio data, first data and second data; fusing the first data and the second data to obtain third data; constructing a multi-modal model, wherein an input layer of the multi-modal model is used for receiving the audio data, the first data and the third data, an output of the multi-modal model is an audio residual error vector quantization encoding, a first encoding corresponding to the first data and a second encoding corresponding to the third data, and a cross attention layer of the multi-modal model is used for fusing the first encoding, the second encoding and the audio residual error vector quantization encoding to obtain a weighted audio feature vector. The multi-modal model is trained to obtain an audio generation model. The model can learn the semantics and acoustic characteristics of long text and context, thereby generating audio output which is more natural, pleasant, more in line with internal semantics and auditory perception characteristics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to a training method and device of an audio generation model, an apparatus and a storage medium. BACKGROUND

[0002] Speech generation technology is being widely applied to short video, robot, audio reading, map navigation, live commentary and other scenarios. At present, in the customer service of the financial field such as bank, insurance and securities, AI agents widely adopt speech generation technology to generate dialogue voice, complete question and answer interaction, notification reminder, marketing recommendation and other tasks.

[0003] In the related art, in the speech generation process, the speech generation is usually performed depending on the volume, tone and other features of the audio data and emotional features, but the combination of the text semantics and context dependence is insufficient, especially the processing and speech generation ability for long text is weak, which causes the generated speech rhythm to be flat, and the user experience is poor. SUMMARY

[0004] The training method and device of the audio generation model, the computer and the storage medium disclosed by the present application solve the problem of insufficient combination of text semantics and context dependence in the existing speech generation technology, and the problem of flat rhythm of the generated speech.

[0005] In a first aspect, the present application provides a training method of an audio generation model, comprising:

[0006] obtaining training data, the training data comprising audio data, first data and second data, the first data being speech transcription text data and music sound effect data corresponding to the audio data, and the second data being description text data corresponding to the audio data;

[0007] fusing the first data and the second data to obtain third data;

[0008] constructing a multi-modal model, the multi-modal model comprising at least an input layer, an intermediate layer and an output layer, the input layer comprising a pre-trained large language model and a pre-trained audio encoder, the output layer comprising an audio decoder, the intermediate layer comprising at least one cross-attention layer; the pre-trained large language model is configured to receive the first data and the third data and encode to obtain a first encoding corresponding to the first data and a second encoding corresponding to the third data; the pre-trained audio encoder is configured to receive the audio data and encode to obtain an audio residual vector quantization encoding; the cross-attention layer is configured to perform cross-attention calculation by taking the second encoding as a query vector, the first encoding as a key vector or a value vector, and the audio residual vector quantization encoding as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization encoding as a key vector or a value vector; and the output layer is configured to receive the weighted audio feature vector and generate an audio;

[0009] training the multi-modal model according to the audio data, the first data and the third data to obtain an audio generation model.

[0010] In a second aspect, the present application provides an audio generation device, comprising:

[0011] a data acquisition module configured to acquire training data, the training data comprising audio data, first data and second data, the first data being speech transcription text data and music sound effect data corresponding to the audio data, and the second data being description text data corresponding to the audio data;

[0012] a data fusion module configured to fuse the first data and the second data to obtain third data;

[0013] a model construction module configured to construct a multi-modal model, the multi-modal model comprising at least an input layer, an intermediate layer and an output layer, the input layer comprising a pre-trained large language model and a pre-trained audio encoder, the output layer comprising an audio decoder, and the intermediate layer comprising at least one cross-attention layer; the pre-trained large language model is configured to receive the first data and the third data and encode to obtain a first encoding corresponding to the first data and a second encoding corresponding to the third data; the pre-trained audio encoder is configured to receive the audio data and encode to obtain an audio residual vector quantization encoding; the cross-attention layer is configured to perform cross-attention calculation by taking the second encoding as a query vector, the first encoding as a key vector or a value vector, and the audio residual vector quantization encoding as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization encoding as a key vector or a value vector; and the output layer is configured to receive the weighted audio feature vector and generate an audio;

[0014] a model training module configured to train a multi-modal model according to the audio data, the first data, and the third data, to obtain an audio generation model.

[0015] In a third aspect, the present application provides a computer device, which comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, and the memory stores a policy model, wherein the computer program, when executed by the processor, implements the method provided in any one of the embodiments of the present application.

[0016] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, causes the processor to implement the method provided in any one of the embodiments of the present application.

[0017] The training method, device, computer and storage medium of the audio generation model, the method comprises: obtaining training data, the training data comprises audio data, first data and second data; the first data and the second data are fused to obtain third data; a multi-modal model with a new architecture is constructed; the output layer is used to receive a weighted audio feature vector to generate an audio; the multi-modal model is trained according to the audio data, the first data and the third data to obtain an audio generation model. On the one hand, since the description text data corresponding to the audio data is used, the model can learn the specific mode of the psychoacoustic features of the second data in the audio in addition to the audio basic features of the first data, so as to generate more natural and more human auditory perception audio; on the other hand, the audio generation model fuses the ability of the pre-trained large language model to extract long text and context semantic features and the ability of the pre-trained audio encoder to extract acoustic features and other features through the cross attention mechanism, improves the recognition ability of the existing audio generation model to long text and context semantic features, and generates more natural and more semantic audio.

[0018] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a step schematic flowchart of the training method of the audio generation model provided by an embodiment of the present application;

[0021] Figure 2 is a framework structure schematic diagram of a multi-modal model provided by an embodiment of the present application;

[0022] Figure 3 is a step schematic flow chart of a residual vector quantization encoding method provided by an embodiment of the present application;

[0023] Figure 4 is a step schematic flow chart of a multi-modal model training method provided by an embodiment of the present application;

[0024] Figure 5 is a structure schematic diagram of an audio generation device provided by an embodiment of the present application;

[0025] Figure 6 is a structure schematic diagram of a computer device provided by an embodiment of the present application.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0028] The flow charts shown in the drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the described order. For example, some operations / steps can be further divided, combined or partially merged, so the actual execution order may be changed according to the actual situation.

[0029] It should be understood that the terms used in this specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, unless otherwise clearly indicated by the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0030] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", etc. are used to distinguish the same or similar items with basically the same function and role. For example, the first data and the second data are only used to distinguish different data, and do not limit the order. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. also do not necessarily mean different.

[0031] It should also be understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0032] In order to facilitate the understanding of the embodiments of the present application, some glossaries involved in the embodiments of the present application are briefly explained as follows.

[0033] 1. Speech transcription text data: Speech transcription text data is data that transcribes spoken information in audio into text form.

[0034] 2. Music sound effect data: The record of music or sound effect audio, which can be used for computer reading, storage, editing, processing, performance or generation. Common forms include MIDI (Musical Instrument Digital Interface) and MusicXML.

[0035] 3. Description text data: Feature data obtained by extracting or labeling audio data, which can involve one or more acoustic features, including loudness, pitch (tone), timbre, aftertaste, nonlinearity, masking effect, binaural effect, emotional feature, phoneme feature.

[0036] 4. Large language model (LLM): Also known as large language model, it refers to a computer model trained on a large amount of text data with advanced text understanding and generation capabilities. These models can be used for various language tasks, including text summarization, translation, question answering, etc.

[0037] 5. Transformer model: A neural network model in the field of natural language processing (NLP), which performs well in tasks such as machine translation, text generation, sentiment analysis, and topic classification.

[0038] The basic architecture of the Transformer model includes an encoder and a decoder, which uses a multi-head self-attention mechanism to process input data in parallel, allowing the model to focus on all positions in the sequence simultaneously during encoding and decoding, thereby capturing long-distance dependencies.

[0039] 6. T5 Large Language Model: A large language model based on the Transformer architecture, capable of handling various text-to-text transfer tasks. T5 stands for "Text-to-Text Transfer Transformer," which learns rich language representations through a pre-training phase, allowing the model to adapt to specific tasks during fine-tuning and achieve excellent performance on multiple downstream tasks such as machine translation, document summarization, question answering, and classification tasks like sentiment analysis.

[0040] 7. GPT-NeoX-20B: A pre-trained large language model based on the Decoder-Only architecture, whose weights are open-source.

[0041] 8. SoundStream: An end-to-end neural audio encoder that uses deep learning techniques to compress and reconstruct audio signals. The core idea of SoundStream is to use neural network models to learn efficient audio representations, achieving high-quality audio compression.

[0042] 9. EmoNet: An artificial intelligence model specifically designed for audio emotion feature extraction, typically based on deep learning techniques such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), or Transformers.

[0043] 10. Decoder-Only Architecture: A neural network design pattern that contains only a Decoder component, without an independent Encoder.

[0044] 11. Mel Frequency Cepstral Coefficients (MFCCs): An audio feature extraction method or form based on the human auditory system, widely used in speech recognition, audio processing, and music information retrieval. Mel Frequency Cepstral Coefficients are obtained by a series of processing steps on audio signals, including pre-emphasis, framing, Fourier transform, Mel filter bank, discrete cosine transform, etc.

[0045] 12. Cross-Attention Mechanism: A variant of self-attention that allows the model to focus on relevant features from another modality (e.g., images) while processing one modality (e.g., text). This mechanism is particularly important in multi-modal tasks as it enhances the interaction of information between different modalities.

[0046] 13. MLM (Masked Language Model): MLM is a training task that focuses on learning language representations at the level of words and phrases. The core idea of MLM is to randomly replace some tokens in the input text with a special [MASK] token, and then let the model predict these masked tokens, thereby training the model to be sensitive to context.

[0047] For example, during the pre-training process of a certain neural network model, MLM is implemented through the following steps:

[0048] Data preparation: Randomly select sentences from a large corpus of text and replace a certain proportion of tokens with the [MASK] token.

[0049] Model input: Input the sentence containing the [MASK] token into the language model, which needs to predict these masked tokens based on the context.

[0050] Loss calculation: Calculate the difference between the predicted tokens and the actual tokens, usually using the cross-entropy loss function.

[0051] Backpropagation: Adjust the model's parameters through the backpropagation algorithm based on the results of the loss function to minimize prediction errors.

[0052] Iterative training: Repeat the above process until the model's performance on the MLM task reaches a satisfactory level.

[0053] 14. NSP (Next Sentence Prediction): NSP is a training task that focuses on learning the coherence and logical relationship between sentences and paragraphs. This task aims to train the model to predict whether the given two sentences are sequentially appearing, i.e., whether the second sentence is a natural continuation of the first sentence.

[0054] NSP encourages the model to learn the logical and semantic relationship between sentences, so that the model can better capture the relationship between texts when dealing with downstream tasks that require understanding of text coherence, such as question answering and natural language inference. For example, during the pre-training process of the T5 model, NSP is implemented through the following steps:

[0055] Data preparation: Randomly select sentence pairs from a large corpus of text. If the second sentence is the direct continuation of the first sentence, it is a positive example; if the second sentence has no direct coherence with the first sentence, it is a negative example.

[0056] Model input: Input the sentence pair as input, and the T5 model will receive the two sentences and process them through its encoder layer.

[0057] Loss Calculation: The model predicts whether a given sentence pair is a "positive example" or a "negative example" and uses a cross-entropy loss function to measure the difference between the model's prediction and the actual label.

[0058] Backpropagation: Based on the results of the loss function, the parameters of the model are adjusted through the backpropagation algorithm to minimize the prediction error.

[0059] Iterative Training: Repeat the above process until the performance of the model on the NSP task reaches a satisfactory level.

[0060] 15. Residual Vector Quantization (RVQ): A technique for data compression. Especially when dealing with large-scale datasets, RVQ can approximate the quantization error of high-dimensional vectors by additional quantizers, gradually refining the quantization results through multiple low-complexity quantizers instead of relying on a high-complexity quantizer, while maintaining low storage and computational requirements while generating effective encoding.

[0061] 16. Generalized Residual Vector Quantization (GRVQ): GRVQ is an improvement over RVQ, whose core idea is to iteratively select a codebook and optimize it, then re-quantize the dataset to obtain new residual vectors for use in the next iteration.

[0062] 17. Codebook: A codebook is a set of pre-defined discrete vectors, or a set of codewords, used to approximate the original continuous feature vectors during the quantization process. A good codebook design should ensure that the distance between codewords is as large as possible to reduce quantization error, and should be optimized according to the statistical properties of audio data.

[0063] 18. Audio Encoder: In a multi-modal model, the audio encoder can be considered as a component that converts speech information into audio feature representations. Possible audio feature representations include Mel-spectrogram, Linear Spectral Pair (LSP), Mel Frequency Cepstral Coefficients (MFCCs), etc., which can capture the acoustic characteristics of speech, such as pitch, timbre, speech rate, etc.

[0064] 19. Tokenizer: Tokenizer is an important tool in text processing, which is responsible for cutting the input continuous text string into smaller tokens, which can be words, subwords, characters or symbols, etc. The design of tokenizer depends on the specific language, model requirements and application scenarios, aiming to improve the efficiency and accuracy of subsequent text processing tasks (such as text classification, sentiment analysis, TTS, etc.). In some embodiments, tokenizer can integrate components such as chunker, tokenization encoder, etc. to realize all preprocessing functions such as chunking, tokenization, tokenization and encoding before the text input large model.

[0065] 20. Prompt Engineering: Also known as prompt engineering, prompt engineering refers to the technology of designing carefully designed text prompts to guide the model to perform specific tasks. These prompts can include instructions, context, examples and other elements, aiming to make the model better understand the task requirements and produce high-quality answers or generated text. The key to prompt engineering is to design effective prompts so that the model can correctly understand the task and give appropriate responses.

[0066] 21. Chain-of-Thought (CoT) Prompting: Also known as thought chain prompting, a technology that uses natural language prompts to guide the model to generate intermediate thinking steps to solve problems. This technology helps the model demonstrate the ability to solve complex problems by adding a series of step-by-step reasoning steps in the prompt, improving the transparency, interpretability of the model output and the generalization ability of the model, and helping to find and correct errors by viewing the steps.

[0067] 22. Instruction and reasoning task dataset: A dataset for instruction and reasoning task, which can enhance the model's response ability and reasoning ability to instructions, including Muffin3, T0-SF and datasets containing Chain of Thought (CoT) annotations.

[0068] 23. Adam (Adaptive Moment Estimation): An optimization algorithm based on gradient, used to solve the first-order optimization problem of random objective function. ADAM algorithm uses the adaptive estimation of the first and second moments of the gradient to provide personalized learning rate for different parameters. This method is easy to implement, high in computational efficiency, small in memory requirement, and particularly suitable for problems with large data volume or high parameter dimension.

[0069] 24. AdamW (Adam Weight Decay Fix): An optimization algorithm widely used in deep learning, which is an improved version based on the Adam optimizer. The main contribution of AdamW lies in its handling of weight decay. In the traditional Adam optimizer, weight decay is added as a regularization term to the loss function, while in AdamW, weight decay is directly applied to the parameter update step, thereby more effectively controlling the complexity of the model and helping to improve the generalization ability of the model.

[0070] 25. Retrieval-Augmented Generation (RAG): An artificial intelligence technology that combines retrieval and generation. The basic principle is to first use a retrieval system to find the most relevant information from a large amount of data according to the user's query. This information is then used as context for the generation model, helping the model generate more accurate and targeted responses.

[0071] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following examples and features in the examples can be combined with each other without conflict.

[0072] In the field of audio generation, especially in speech generation technology, existing machine learning methods usually focus on extracting physical acoustic features of audio, such as Mel-frequency cepstral coefficients, fundamental frequency (F0), hidden Markov model states, etc. These features help machine learning models learn the physical acoustic features of audio.

[0073] However, compared with existing acoustic sensors such as microphones, the human auditory organ has the characteristics of nonlinearity, difficulty in quantification, physiology, and complexity, making the human auditory system more delicate and complex when processing sound. Therefore, the human auditory system does not directly perceive these physical acoustic features, but perceives more complex psychoacoustic features, such as loudness, pitch, timbre, masking effect, residual tone, binaural effect, etc. Moreover, the psychoacoustic features of speech often interact with the semantic features and specific structures of language, so the speech generated by simply combining phoneme features and emotion features is not natural enough. Incorporating physical properties, psychoacoustic features, and semantic features into the training of audio generation models can enable the model to learn audio features that are more consistent with human auditory habits. At the same time, by integrating the long text and context processing capabilities of large language models, the model can learn the inherent pattern structure and semantic connotation of language, thereby generating more natural, pleasant, and semantically and audibly consistent audio output.

[0074] In addition, the diffusion model used by existing multi-modal models has certain value in the field of audio generation, but compared with large language models based on Decoder-Only architecture, the diffusion model has disadvantages in consistency, controllability and response speed of longer speech. In the financial customer service voice interaction scene, the customer service needs to use different rhetoric under the guidance of professional and rigorous specifications, and at the same time, as much as possible, more efficient and fast in the dialogue to solve the difficulties for the customer. If the consistency and controllability are not enough, the customer service reply may cause negative business and legal consequences. If the response speed is not enough, the customer will lose patience and have a disappointed emotion for the AI seat service, and directly choose manual service.

[0075] Compared with using a diffusion model, the advantages of using a large language model based on Decoder-Only architecture are that it makes it possible to improve the rule fidelity, instruction fidelity and controllability of the subsequent audio generation model based on RAG technology, and can make full use of self-owned data and link the existing business and research and development process system. The self-owned data can include a rhetoric library, regulations, a music library, an audio material library, etc., further enhancing the instruction following ability of the audio generation model and the quality and compliance of the generated audio.

[0076] To solve the above problems, an embodiment of the present application provides a training method of an audio generation model. Please refer to Figure 1 , Figure 1 is a step schematic flowchart of the training method of the audio generation model provided by an embodiment of the present application.

[0077] As shown in Figure 1 , the training method of the audio generation model specifically includes steps S101 to S104.

[0078] S101, obtaining training data, the training data including audio data, first data and second data, the first data being audio data corresponding speech transcription text data and music audio data, and the second data being audio data corresponding description text data.

[0079] Illustratively, when it is necessary to train an audio generation model for a voice interaction task, the audio data in the training data can be voice data, the first data being voice data corresponding speech transcription text data, and the second data being description text data of the characteristics of the voice data. The characteristics described by the second data can include, for example, loudness, pitch (tone), timbre, after-tone, non-linearity, masking effect, binaural effect and other psychoacoustic characteristics, and can also include, for example, speaker gender, speaker role, style, habit, accent, emotion, double meaning, connotation, essence, dialect and other relatively objective characteristics. The characteristics of the audio data provided by the description text are a beneficial supplement to the physical acoustic characteristics extracted by the machine.

[0080] Exemplarily, when training the audio generation model for a music sound effect generation task, the audio data in the training data is music or sound effect audio, the first data is music sound effect data corresponding to the music or sound effect audio, and the music sound effect data can be a digitalized music score in MIDI format or MusicXML format. These formats can better record the pitch, rhythm, dynamics and other information of music, and provide a structured representation of music for the model. The second data is a description text data of the features of the music data. These descriptions can include descriptions of music style, emotion, instrument type, etc., and can contain music expression instructions such as speed, dynamics, transposition, etc. These descriptive text data helps the model learn the emotional and stylistic features of the music, so as to better capture the expression intention and emotional color of the music when generating music sound effects.

[0081] S102, fuse the first data and the second data to obtain third data.

[0082] Specifically, the fusion manner can be instruction reinforcement or reasoning reinforcement of the first data combined with the second data, or splicing.

[0083] Exemplarily, taking instruction reinforcement as an example and taking the construction of a financial customer service dialogue data set as an example, the first data includes:

[0084] Question: "What is my account balance?"

[0085] Answer: "Your account balance is $X."

[0086] The second data includes:

[0087] The tone of the answer is pleasant, smooth and healthy.

[0088] The third data obtained by fusion includes:

[0089] Instruction: "Please answer the customer's account balance query in the form of 'Your account balance is $X' with a pleasant tone, smooth and healthy tone."

[0090] Question: "What is my account balance?"

[0091] Answer: "Your account balance is $X."

[0092] S103, construct a multi-modal model.

[0093] As shown in Figure 2 , the multi-modal model includes a text encoder, a feature extractor, a feature decoder, and a text decoder. Figure 2is a schematic diagram of a framework structure of a multi-modal model provided by an embodiment of the present application, which is used to perform the training method of the audio generation model. Wherein, the audio generation device can be configured in a terminal or a server. The multi-modal model at least includes an input layer, an intermediate layer and an output layer, the input layer includes a pre-trained large language model and a pre-trained audio encoder, the output layer includes an audio decoder, and the intermediate layer at least includes a cross-attention layer; the pre-trained large language model is used to receive first data and third data, and encode to obtain first encoding corresponding to the first data and second encoding corresponding to the third data; the pre-trained audio encoder is used to receive audio data and encode to obtain audio residual vector quantization encoding; the cross-attention layer is used to perform cross-attention calculation by taking the second encoding as a query vector, taking the first encoding as a key vector or a value vector, and taking the audio residual vector quantization encoding as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization encoding as a key vector or a value vector; and the output layer is used to receive the weighted audio feature vector and generate audio.

[0094] Wherein, the pre-trained large language model is used to receive first data and third data, and encode to obtain first encoding corresponding to the first data and second encoding corresponding to the third data.

[0095] The large language model has good text processing and encoding ability, can effectively extract context dependence and semantic features in long text and long dialogue, and output feature representation in the form of encoding.

[0096] Specifically, in order to convert the first data and the third data into the first encoding and the second encoding for training the multi-modal model, the text preprocessing needs to be performed according to the functional integration of the text preprocessing part of the pre-trained large language model, such as block division, word segmentation, tokenization, or using some modules in the large language model for text preprocessing, or directly inputting the text into the large language model to obtain the first encoding and the second encoding.

[0097] For example, the first data and the third data are input into a word segmenter respectively to obtain tokenization information corresponding to the first data and the third data. If GPT-NeoX-20B based on Decoder-Only architecture is used as the pre-trained large language model of the input layer of the multi-modal model to encode the input text, the following steps are included:

[0098] Step one: in the case that the text length does not exceed the model limit and does not need to be divided into blocks, a tokenization encoder such as AutoTokenizer.from_pretrained('EleutherAI / gpt-neox-20b') is used to receive the input text to obtain tokenization information.

[0099] Step two: using the GPT-NeoX-20B model, such as AutoModelForCausalLM.from_pretrained('EleutherAI / gpt-neox-20b'), receiving tokenization information to obtain semantic representation.

[0100] For the GPT-NeoX-20B model, the last token of the hidden state H(T) of the last layer of the GPT-NeoX-20B model can be selected as the semantic representation E(T) of the whole text, and the expression is as follows:

[0101] E(T) = H(T)[-1]

[0102] Exemplarily, the T5 large language model based on the Encoder-Decoder architecture can also be used as the pre-training large language model of the input layer of the multi-modal model, and the input text is encoded, which includes the following steps:

[0103] Step one: the input text is divided into blocks by sliding window, and the blocked text is obtained.

[0104] Step two: using the tokenizer and encoder, such as T5Tokenizer.from_pretrained('t5-large'), receiving the blocked text to obtain tokenization information.

[0105] Step three: using the T5 large language model, such as T5Model.from_pretrained('t5-large'), receiving tokenization information to obtain semantic representation.

[0106] It should be noted that for the T5 large language model, the first token of the hidden state H(T) of the last layer of the T5 large language model corresponding to each piece of text data is usually selected as the semantic representation E(T) of the whole text, and the expression is as follows:

[0107] E(T) = H(T)[0]

[0108] It should be noted that in step S103, the semantic representation output by the pre-training large language model of the input layer of the multi-modal model can be selected according to the architecture of the model or the richness of the semantic features contained.

[0109] It should be noted that in step S103, if the text data is blocked, the audio data can be aligned and blocked, so that the audio data used for training is aligned with the blocked text data.

[0110] Exemplarily, in step S103, the pre-trained audio encoder is used to receive the audio data and encode the audio data to obtain an audio residual vector quantization encoding. In an example, the pre-trained audio encoder can be a SoundStream audio encoding model, which is configured to perform residual vector quantization encoding on the audio data. The SoundStream audio encoding model includes the following steps:

[0111] Step 1: input the audio data x into the encoding layer of the audio encoding model to obtain a feature vector f(x) of the audio data, which is initialized as a residual q = f(x).

[0112] Step 2: input the obtained feature vector f(x) into the quantizer layer, which maps the continuous feature vector to a discrete index space to obtain a set of discrete indexes q. The quantization process can be represented as:

[0113] q = Q(f(x)) = arg min c∈C ||f(x) - c||

[0114] where Q is a quantization function, ||·|| represents the Euclidean distance, and c is an element in a predefined codebook C.

[0115] Step 3: use the pre-trained decoder D to reconstruct the audio feature from the current residual r to obtain a new residual r new :

[0116]

[0117] where

[0118] Step 4: repeat steps 2 and 3 until the residual r t is less than a threshold ∈ or reaches a predetermined number of iterations T, which can be represented as:

[0119]

[0120] Step 5: quantize the final residual r T again and save the residual as an audio residual vector quantization encoding.

[0121] In some embodiments, the pre-trained audio encoder includes an audio residual vector quantization encoder, an emotion feature extractor, and a feature fusioner. Please refer to Figure 3 , Figure 3is a step schematic flow chart of a residual vector quantization encoding method provided by an embodiment of the present application. Take the SoundStream model as an audio encoding model of a pre-trained audio encoder, take the EmoNet model as an emotional feature extractor of a pre-trained audio encoder to perform audio emotional feature extraction, construct a feature fusion device for feature fusion by splicing or attention mechanism, as shown in Figure 3 The residual vector quantization encoding of the audio is specifically composed of the following steps:

[0122] S103a, convert the audio data into a spectrogram or mel-frequency cepstral coefficient to obtain preprocessed audio data x.

[0123] S103b, input the preprocessed audio data x into the audio encoding model to obtain the basic feature f base (x), whose expression is as follows:

[0124] f base (x)=SoundStream(x)

[0125] S103c, input the preprocessed audio data x into the emotional feature extractor to obtain the emotional feature f emo (x), whose expression is as follows:

[0126] f emo (x)=EmoNet(x)

[0127] S103d, fuse the basic feature and the emotional feature by splicing or attention mechanism to obtain a fused feature representation, whose expression is as follows:

[0128] f fused (x)=Concat(f base (x),f emo (x))

[0129] S103e, input the fused feature representation into the audio encoding model to generate the residual vector quantization encoding q, whose expression is as follows:

[0130] q=Q(f fused (x))

[0131] Wherein, Q is a quantization function.

[0132] For example, Figure 2As shown, the multimodal model may include an input layer, an intermediate layer, and an output layer. The input layer may include the pre-trained audio encoder and the pre-trained large language model. The intermediate layer may include a cross-attention layer and a large language model based on a decoder-only architecture containing the cross-attention layer. Alternatively, the intermediate layer may include a cross-attention layer and a large language model based on a decoder-only architecture concatenated with the cross-attention layer. The output layer may include an audio decoder and a vocoder concatenated with the audio decoder. The vocoder may be part of the audio decoder or may not be included within the multimodal large model.

[0133] The input layer of this multimodal model receives the data to be trained, encodes it, and outputs the first encoding, the second encoding, and the audio residual vector quantized encoding to the intermediate layer.

[0134] The input layer of this multimodal model receives training data including audio data, first data, and second data. The first data consists of speech-transcribed text data and music sound effect data corresponding to the audio data, and the second data consists of descriptive text data corresponding to the audio data.

[0135] In some embodiments, the input layer of the multimodal model receives and encodes the data to be trained, and outputs the first encoding, the second encoding, and the audio residual vector quantization encoding to the intermediate layer. Specifically, this includes: fusing the first data and the second data to obtain the third data; processing the audio data using a pre-trained audio encoder to obtain the audio residual vector quantization encoding; processing the first data and the third data using a pre-trained large language model to obtain the first encoding corresponding to the first data and the second encoding corresponding to the third data; and outputting the first encoding, the second encoding, and the audio residual vector quantization encoding to the intermediate layer.

[0136] In other embodiments, the input layer of the multimodal model receives and encodes the data to be trained, and after performing feature preprocessing operations on the data of different modalities, outputs the first encoding, the second encoding, and the audio residual vector quantization encoding to the intermediate layer. Specifically, this includes: processing audio data using a pre-trained audio encoder to obtain audio residual vector quantization encoding; processing the first data and the third data using a pre-trained large language model to obtain the first encoding corresponding to the first data and the second encoding corresponding to the third data; performing feature preprocessing operations such as pooling, linearization, and data alignment on the first encoding, the second encoding, and the audio residual vector quantization encoding; and outputting the first encoding, the second encoding, and the audio residual vector quantization encoding to the intermediate layer. The pre-trained audio encoder is used to process audio data to obtain audio residual vector quantization encoding; the pre-trained large language model is used to process the first data and the third data to obtain the first encoding and the second encoding.

[0137] In some embodiments, the input layer of the multi-modal model receives the training data and encodes the training data, and outputs the first encoding, the second encoding, and the audio residual vector quantization encoding to the intermediate layer. The intermediate layer performs feature preprocessing on the first encoding, the second encoding, and the audio residual vector quantization encoding, and performs cross-attention calculation. Specifically, if the cross-attention layer of the intermediate layer is part of the large language model based on the Decoder-Only architecture, the first encoding, the second encoding, and the audio residual vector quantization encoding can be subjected to feature preprocessing operations such as pooling, linearization, and data alignment by the large language model based on the Decoder-Only architecture, or by other components of the intermediate layer. The other components of the intermediate layer can include a linear layer or a linear projection layer.

[0138] The intermediate layer is used to fuse the audio residual vector quantization encoding with the first encoding and the second encoding to generate expected audio feature expression. The cross-attention layer uses the second encoding as a query vector, and uses the first encoding as a key vector or a value vector, and uses the audio residual vector quantization encoding as a value vector or a key vector to perform cross-attention calculation to obtain cross-attention layer weights.

[0139] For example, the second encoding is used as a query vector Q, the first encoding is used as a key vector K, and the audio residual vector quantization encoding is used as a value vector V to perform cross-attention calculation, which includes the following steps:

[0140] Step one: length alignment of the training data is performed by using linear interpolation or pooling operation to realize length alignment of the audio residual vector quantization encoding, the first encoding, and the second encoding, and to obtain length-aligned training data.

[0141] Step two: dimension alignment of the length-aligned training data is performed by using linear transformation or projection to obtain dimension-aligned training data.

[0142] Step three: before cross-attention calculation, the training data of different modalities can be preliminarily fused by splicing or weighted summation operation, or the correlation of the training data of different modalities can be guided in advance by using attention guidance method to obtain processed training data.

[0143] Step four: the first encoding is used as a key vector K, the second encoding is used as a query vector Q, and the audio residual vector quantization encoding is used as a value vector V to perform cross-attention calculation. The corresponding calculation formula is:

[0144] The attention score S is calculated as follows:

[0145]

[0146] The attention weights A are calculated,

[0147] A = softmax(S)

[0148] The weighted sum is calculated,

[0149] O = AV

[0150] The weighted audio residual vector output O is obtained.

[0151] If the large language model based on Decoder-Only architecture included in the intermediate layer is in series with the cross-attention layer, the large language model based on Decoder-Only architecture is used to receive the weighted audio residual vector of the cross-attention layer, to generate the expected audio feature expression, which can be in the form of a spectrogram or a mel-frequency cepstral coefficient, or an implicit representation of the expected audio.

[0152] If the cross-attention layer included in the intermediate layer is part of the large language model based on Decoder-Only architecture, the large language model based on Decoder-Only architecture is used to receive the audio residual vector quantization encoding and the first encoding and the second encoding, to generate the expected audio feature expression.

[0153] The output layer is used to receive the expected audio feature expression generated by the intermediate layer, to generate an audio expression form such as a mel-frequency cepstral coefficient or to reconstruct an audible audio waveform.

[0154] For example, the output layer includes an audio decoder and a vocoder, wherein the audio decoder is used to receive the implicit representation of the expected audio generated by the intermediate layer to obtain the mel-frequency cepstral coefficient of the expected audio; and the vocoder is used to receive the mel-frequency cepstral coefficient of the expected audio to obtain an audible audio signal.

[0155] For example, the output layer includes an audio decoder and a vocoder, wherein the audio decoder is used to receive the residual vector quantization encoding of the expected audio generated by the intermediate layer to obtain the mel-frequency cepstral coefficient of the expected audio; and the vocoder is used to receive the mel-frequency cepstral coefficient of the expected audio to obtain an audible audio signal.

[0156] For example, the output layer only includes an audio decoder, which is used to receive the implicit representation of the expected audio generated by the intermediate layer to obtain the mel-frequency cepstral coefficient of the expected audio.

[0157] The audio decoder is configured to receive the expected audio feature representation generated by the intermediate layer, and generate a mel-frequency cepstral coefficient or other explicit feature representation of the audio.

[0158] S104, training the multi-modal model according to the audio data, the first data and the third data to obtain an audio generation model.

[0159] Specifically, the parameters of the pre-trained audio encoder of the input layer of the multi-modal model and the pre-trained large language model of the input layer can be frozen; the parameters of at least one layer of the audio decoder of the intermediate layer or the output layer can be frozen; the audio data, the first data and the third data input into the multi-modal model are used to calculate a loss by using an NSP method and / or an MLM method; the unfrozen parameters of the multi-modal model are updated according to the loss by using an AdamW optimizer; the two steps of using the audio data, the first data and the third data input into the multi-modal model to calculate a loss by using an NSP method and / or an MLM method, and updating the unfrozen parameters of the multi-modal model according to the loss by using an AdamW optimizer are repeated until the loss is less than a threshold value or repeated a certain number of times, to obtain an audio generation model.

[0160] In some embodiments, the training of the multi-modal model according to the audio data, the first data and the third data specifically comprises the following steps:

[0161] The parameters of the pre-trained audio encoder of the input layer of the multi-modal model and the pre-trained large language model of the input layer are frozen; if the intermediate layer does not include a Decoder-Only based large language model, the parameters of several layers of the audio decoder of the output layer are frozen; if the intermediate layer includes a Decoder-Only based large language model, the parameters of several layers of the intermediate layer and the audio decoder can be frozen, or all the parameters of the audio decoder can be frozen to prevent overfitting; the audio data, the first data and the third data input into the multi-modal model are used to calculate a loss by using an NSP method and / or an MLM method, and the calculated loss can include a cross-entropy loss; the unfrozen parameters of the multi-modal model are updated according to the loss by using an AdamW optimizer; the two steps of using the audio data, the first data and the third data input into the multi-modal model to calculate a loss by using an NSP method and / or an MLM method, and updating the unfrozen parameters of the multi-modal model according to the loss by using an AdamW optimizer are repeated until the loss is less than a threshold value or repeated a certain number of times, to obtain an audio generation model.

[0162] The multi-modal model is trained by using an MLM method to obtain an audio generation model, and specifically comprises:

[0163] Masking operation: randomly mask a certain proportion of text vocabulary (usually 15%) and its corresponding audio segment.

[0164] Model input: input the masked text and audio data into the multi-modal model.

[0165] Calculate loss: y t is the audio residual vector quantization encoding at time step t, calculate the cross-entropy loss, get the loss L, that is:

[0166] where T is the sequence length

[0167] The cross-entropy loss between the predicted audio residual vector quantization encoding and the actual audio residual vector quantization encoding can also be calculated, the loss between the expected audio data generated by the prediction and the actual input audio data is calculated, and the two losses are combined by weighting to get the total loss.

[0168] Backpropagation and parameter update: according to the obtained loss, use an optimizer such as AdamW to update the model parameters.

[0169] Exemplary, a large language model based on a Decoder-Only architecture with a pre-trained GPT-NeoX model as an intermediate layer, taking audio residual vector quantization encoding, first encoding, and second encoding as input of the intermediate layer, to get the expected audio residual vector quantization encoding output of the intermediate layer. The step of training the multi-modal model according to the audio data, the first data and the third data includes: freezing the parameters of the pre-trained audio encoder of the input layer and the pre-trained large language model of the input layer, to prevent weakening the feature extraction ability of the input layer neural network model for audio and text. Obtain a pre-trained GPT-NeoX model, freeze part of the layers of the pre-trained GPT-NeoX model to avoid overfitting; train the multi-modal model using the NSP method to obtain an audio generation model,

[0170] wherein the multi-modal model is trained using the NSP method, specifically including:

[0171] Data preparation: construct positive samples corresponding to the third data and audio data and irrelevant negative samples by randomly pairing audio data and third data.

[0172] Model input: input the positive and negative samples into the multi-modal model to get the predicted samples.

[0173] Calculate loss: maximize the probability of correctly predicting positive and negative samples, calculate the cross-entropy loss.

[0174] Backpropagation and parameter update: according to the obtained loss, use an optimizer such as AdamW to update the model parameters.

[0175] For example, refer to Figure 4 , Figure 4 is a step schematic flowchart of a multi-modal model training method provided by an embodiment of the present application. A pre-trained GPT-NeoX model is used as a large language model based on a Decoder-Only architecture, and audio residual vector quantization encoding, first encoding, and second encoding are used as inputs of the intermediate layer to obtain the expected audio residual vector quantization encoding output of the intermediate layer, as shown in Figure 4 S104 specifically includes:

[0176] S104a, freeze the parameters of the pre-trained audio encoder of the input layer and the pre-trained large language model of the input layer to prevent weakening the feature extraction capability of the input layer neural network model for audio and text.

[0177] S104b, freeze the parameters of at least one layer of the audio decoder of the intermediate layer or the output layer, which can include obtaining a pre-trained GPT-NeoX model and freezing a part of the pre-trained GPT-NeoX model to avoid overfitting.

[0178] S104c, the multi-modal model is trained using the NSP method and the MLM method, specifically including alternately or sequentially using the NSP method and the MLM method as described above to calculate the loss according to the audio data, the first data and the third data input to the multi-modal model.

[0179] S104d, update the unfrozen parameters of the multi-modal model based on the AdamW optimizer according to the loss.

[0180] S104e, repeat steps S104c and S104d until the loss is less than a threshold or a certain number of repetitions is reached, to obtain an audio generation model.

[0181] In some embodiments, the model training method further comprises: obtaining generation prompt data, inputting the generation prompt data into the audio generation model to obtain audio corresponding to the generation prompt data, specifically including the following steps:

[0182] Step one: obtain the generation prompt data and input it into the input layer of the multi-modal model; if the generation prompt data is in audio form, input the generation prompt data in audio form into the audio encoder of the input layer of the multi-modal model; if the generation prompt data is in text form, input the generation prompt data in text form into the pre-trained large language model of the input layer of the multi-modal model.

[0183] Step two: the input layer of the multi-modal model receives the generation prompt data, generates audio residual vector quantization encoding, and outputs it to the intermediate layer of the multi-modal model.

[0184] Step 3: The intermediate layer of the multimodal model receives the audio residual vector quantization encoding, generates the desired audio feature representation, and outputs it to the output layer.

[0185] Step 4: The multimodal model output layer receives the desired audio feature representation. If the audio decoder of the multimodal model output layer includes a vocoder function, or if the multimodal model output layer includes a vocoder, then the audio corresponding to the generated prompt data is a generated audible sound. If the audio decoder of the multimodal model output layer does not include a vocoder function, and the multimodal model output layer does not include a vocoder, then the audio corresponding to the generated prompt data is a digital representation of the audio, which may include Mel-spectrogram, linear spectrum pairs (LSP), Mel-frequency cepstral coefficients (MFCCs), etc.

[0186] In some embodiments, the model training method further includes: using the LoRA method to train or fine-tune the multimodal model based on the audio data, the first data, and the third data to obtain an audio generation model.

[0187] For example, if in step S103, when constructing the multimodal large model, a pre-trained GPT-NeoX model is used as the intermediate layer of a Decoder-Only architecture-based large language model, and audio residual vector quantization encoding, first encoding, and second encoding are used as inputs to the intermediate layer to obtain the expected audio residual vector quantization encoding output of the intermediate layer, then step S104 may specifically include: freezing the parameters of the pre-trained GPT-NeoX model and the output layer; inserting LoRA modules before several layers of the intermediate layer and initializing the LoRA weight matrix; alternately using the NSP method and the MLM method for model input and loss calculation of the multimodal model; calculating the gradient based on the loss and updating the parameters of the LoRA layer using an optimizer (such as AdamW); and obtaining the multimodal model.

[0188] like Figure 5 As shown, Figure 5 This is a schematic diagram of an audio generation device provided in an embodiment of this application. The audio generation device is used to execute the aforementioned training method for an audio generation model. The audio generation device can be configured on a terminal or a server.

[0189] like Figure 5 As shown, the audio generation device 100 includes a data acquisition module 101, a data fusion module 102, a model building module 103, and a model training module 104.

[0190] The data acquisition module 101 is configured to acquire training data, the training data comprising audio data, first data and second data, the first data being speech transcription text data and music sound effect data corresponding to the audio data, and the second data being description text data corresponding to the audio data;

[0191] The data fusion module 102 is configured to fuse the first data and the second data to obtain third data.

[0192] The model construction module 103 is configured to construct a multi-modal model, the multi-modal model comprising at least an input layer, an intermediate layer and an output layer, the input layer comprising a pre-trained large language model and a pre-trained audio encoder, the output layer comprising an audio decoder, and the intermediate layer comprising at least one cross-attention layer; the pre-trained large language model is configured to receive the first data and the third data and encode the first data and the third data to obtain first encoding corresponding to the first data and second encoding corresponding to the third data; the pre-trained audio encoder is configured to receive the audio data and encode the audio data to obtain audio residual vector quantization encoding; the cross-attention layer is configured to perform cross-attention calculation by taking the second encoding as a query vector, the first encoding as a key vector or a value vector, and the audio residual vector quantization encoding as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization encoding as a key vector or a value vector; and the output layer is configured to receive the weighted audio feature vector and generate audio.

[0193] The model training module 104 is configured to train the multi-modal model according to the audio data, the first data and the third data to obtain an audio generation model.

[0194] It should be noted that, for the convenience and brevity of description, the specific working processes of the above-described device and modules can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0195] The above-described device can be implemented in the form of a computer program, which can run on a computer device as shown in Figure 6 .

[0196] Please refer to Figure 6 , Figure 6 is a structural schematic block diagram of a computer device provided by an embodiment of the present application. The computer device can be a server. Please refer to Figure 6 , the computer device comprises a processor, a memory and a network interface connected through a system bus, wherein the memory can comprise a non-volatile storage medium and an internal memory.

[0197] The non-volatile storage medium can store an operating system and a computer program. The computer program comprises program instructions, which, when executed, can cause the processor to perform any one of the training methods of the audio generation model.

[0198] The processor is configured to provide computing and control capabilities to support the operation of the entire computer device.

[0199] The internal memory provides an environment for the running of a computer program in a non-volatile storage medium, which, when executed by the processor, can cause the processor to execute any one of the training methods of the audio generation model.

[0200] The network interface is configured to perform network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0201] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0202] In one embodiment, the processor is configured to run a computer program stored in the memory to implement the following steps:

[0203] Obtain the training data, the training data includes audio data, first data and second data, the first data is the speech transcription text data and music sound effect data corresponding to the audio data, and the second data is the description text data corresponding to the audio data;

[0204] Fuse the first data and the second data to obtain third data;

[0205] constructing a multi-modal model, the multi-modal model comprising at least an input layer, an intermediate layer, and an output layer, the input layer comprising a pre-trained large language model and a pre-trained audio encoder, the output layer comprising an audio decoder, the intermediate layer comprising at least a cross-attention layer; the pre-trained large language model is configured to receive the first data and the third data, and encode the first data and the third data to obtain a first encoding corresponding to the first data and a second encoding corresponding to the third data; the pre-trained audio encoder is configured to receive the audio data, and encode the audio data to obtain an audio residual vector quantization encoding; the cross-attention layer is configured to perform cross-attention calculation by taking the second encoding as a query vector, the first encoding as a key vector or a value vector, and the audio residual vector quantization encoding as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization encoding as a key vector or a value vector; and the output layer is configured to receive the weighted audio feature vector, and generate the audio.

[0206] training the multi-modal model according to the audio data, the first data, and the third data to obtain an audio generation model.

[0207] In some embodiments, before the multi-modal model is constructed, the processor is further configured to implement: inputting the first data and the third data into a tokenizer respectively, to obtain tokenization information corresponding to the first data and the third data. The pre-trained large language model is configured to receive the first data and the third data, and encode the first data and the third data to obtain a first encoding corresponding to the first data and a second encoding corresponding to the third data, and further comprising: the pre-trained large language model is configured to receive the tokenization information corresponding to the first data and the third data, and encode the first data and the third data to obtain a first encoding corresponding to the first data and a second encoding corresponding to the third data.

[0208] In some embodiments, the intermediate layer comprises a cross-attention layer and a large language model based on a Decoder-Only architecture, and the output layer comprises an audio decoder and a vocoder; the cross-attention layer is configured to perform cross-attention calculation by taking the second encoding as a query vector, the first encoding as a key vector or a value vector, and the audio residual vector quantization encoding as a value vector or a key vector, to obtain a weighted audio residual vector quantization encoding corresponding to the audio residual vector quantization encoding as a key vector or a value vector; the large language model based on the Decoder-Only architecture is configured to utilize the weighted audio residual vector quantization encoding, and infer to obtain an expected audio residual vector quantization encoding; the audio decoder is configured to receive the expected audio residual vector quantization encoding, and decode the expected audio residual vector quantization encoding to obtain an expected audio feature representation, the expected audio feature representation comprising mel-frequency cepstral coefficients; and the vocoder is configured to receive the expected audio feature representation, and generate the audio.

[0209] In some embodiments, the processor is configured to implement, in the training of the multi-modal model according to the audio data, the first data and the third data:

[0210] freezing parameters of a pre-trained audio encoder of an input layer of the multi-modal model and a pre-trained large language model of the input layer;

[0211] freezing parameters of at least one layer of an audio decoder of the intermediate layer or the output layer;

[0212] inputting the audio data, the first data and the third data into the multi-modal model using the NSP method and the MLM method and calculating a loss;

[0213] updating the unfrozen parameters of the multi-modal model based on the AdamW optimizer according to the loss; repeating the two steps of inputting the audio data, the first data and the third data into the multi-modal model based on the NSP method and the MLM method and calculating a loss and updating the unfrozen parameters of the multi-modal model based on the AdamW optimizer according to the loss until the loss is less than a threshold value or the repetition reaches a certain number of times, to obtain an audio generation model.

[0214] In some embodiments, the multi-modal model comprises at least an input layer, an intermediate layer and an output layer, the input layer comprising a pre-trained large language model and a pre-trained audio encoder, the pre-trained audio encoder comprising an audio residual vector quantization encoder, an emotion feature extractor and a feature fusioner; the audio residual vector quantization encoder is configured to receive the audio data and obtain a basic feature based on a residual vector quantization or generalized residual vector quantization method; the emotion feature extractor is configured to receive the audio data and perform emotion feature extraction to obtain an emotion feature; the feature fusioner fuses the basic feature and the emotion feature based on a splicing or attention mechanism method to obtain a fused feature representation; the audio residual vector quantization encoder is configured to receive the fused feature representation and obtain the audio residual vector quantization encoding based on a residual vector quantization or generalized residual vector quantization method.

[0215] In some embodiments, the processor is further configured to implement: obtaining generation prompt data, inputting the generation prompt data into the audio generation model to obtain audio corresponding to the generation prompt data.

[0216] In some embodiments, the processor is further configured to implement: training or fine-tuning a multi-modal model according to the audio data, the first data and the third data using a LoRA method to obtain an audio generation model.

[0217] The embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program comprises program instructions, and the processor executes the program instructions to realize the training method of any one of the audio generation models provided by the embodiment of the present application.

[0218] The computer readable storage medium can be an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0219] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any modification or replacement within the technical range disclosed by the present application can be easily thought of by those skilled in the art, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A training method for an audio generation model, comprising: Acquire training data, which includes audio data, first data, and second data. The first data consists of speech transcription text data and music sound effect data corresponding to the audio data, and the second data consists of descriptive text data corresponding to the audio data. The first and second data are merged to obtain the third data; Construct a multimodal model, which includes at least an input layer, an intermediate layer, and an output layer. The input layer includes a pre-trained large language model and a pre-trained audio encoder. The output layer includes an audio decoder. The intermediate layer includes at least one cross-attention layer. The pre-trained large language model is used to receive the first data and the third data, and encode them to obtain a first code corresponding to the first data and a second code corresponding to the third data; the pre-trained audio encoder is used to receive the audio data and encode it to obtain an audio residual vector quantization code; the cross-attention layer is used to perform cross-attention calculation using the second code as a query vector, the first code as a key vector or value vector, and the audio residual vector quantization code as a value vector or key vector to obtain a weighted audio feature vector corresponding to the audio residual vector quantization code as a key vector or value vector; the output layer is used to receive the weighted audio feature vector and generate audio. The multimodal model is trained based on the audio data, the first data, and the third data to obtain an audio generation model.

2. The method according to claim 1, characterized in that, Prior to constructing the multimodal model, the following is also included: The first data and the third data are respectively input into the word segmenter to obtain the word segmentation information corresponding to the first data and the third data; The pre-trained large language model is used to receive the first data and the third data, and encode them to obtain a first code corresponding to the first data and a second code corresponding to the third data. It also includes: the pre-trained large language model is used to receive lexicalization information corresponding to the first data and the third data, and encode it to obtain a first code corresponding to the first data and a second code corresponding to the third data.

3. The method according to claim 1, characterized in that, The intermediate layer of the multimodal model includes a cross-attention layer and a large language model based on a Decoder-Only architecture, and the output layer includes an audio decoder and a vocoder. The cross-attention layer is used to perform cross-attention calculations using the second encoding as the query vector, the first encoding as the key vector or value vector, and the audio residual vector quantization encoding as the value vector or key vector, to obtain the weighted audio residual vector quantization encoding corresponding to the audio residual vector quantization encoding as the key vector or value vector; the large language model based on the Decoder-Only architecture is used to utilize the weighted audio residual vector quantization encoding and infer the desired audio residual vector quantization encoding; the audio decoder is used to receive the desired audio residual vector quantization encoding and decode it to obtain the desired audio feature representation, which includes Mel-frequency cepstral coefficients; the vocoder is used to receive the desired audio feature representation and generate audio.

4. The method according to claim 1, characterized in that, The training of the multimodal model based on the audio data, the first data, and the third data includes: Freeze the parameters of the pre-trained audio encoder of the input layer of the multimodal model and the pre-trained large language model of the input layer; Freeze the parameters of at least one layer of the audio decoder of the intermediate layer or the output layer; The loss is calculated based on the audio data, first data, and third data input to the multimodal model using the NSP method and / or MLM method. The unfrozen parameters of the multimodal model are updated using the AdamW optimizer based on the loss. Repeat the steps of using the NSP method and / or MLM method to calculate the loss based on the audio data, first data, and third data input to the multimodal model, and updating the unfrozen parameters of the multimodal model based on the loss using the AdamW optimizer, until the loss is less than a threshold or the steps are repeated a certain number of times to obtain the audio generation model.

5. The method according to claim 1, characterized in that, The pre-trained audio encoder includes an audio residual vector quantization encoder, an emotion feature extractor, and a feature fusion unit. The audio residual vector quantization encoder is used to receive the audio data and obtain basic features using a residual vector quantization algorithm or a generalized residual vector quantization algorithm. The emotion feature extractor is used to receive the audio data and extract emotion features to obtain emotion features; The feature fusion unit, based on a splicing or attention mechanism, fuses the basic features and the sentiment features to obtain a fused feature representation; The audio residual vector quantization encoder is used to receive the fused feature representation and obtain the audio residual vector quantization code using a residual vector quantization algorithm or a generalized residual vector quantization algorithm.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Obtain the generated prompt data, input the generated prompt data into the audio generation model, and obtain the audio corresponding to the generated prompt data.

7. The method according to any one of claims 1-5, characterized in that, The step of training the multimodal model based on the audio data, the first data, and the third data to obtain the audio generation model includes: Using the LoRA method, the multimodal model is trained or fine-tuned based on the audio data, the first data, and the third data to obtain an audio generation model.

8. An audio generation device, characterized in that, include: The data acquisition module is used to acquire training data, which includes audio data, first data, and second data. The first data is speech transcription text data and music sound effect data corresponding to the audio data, and the second data is descriptive text data corresponding to the audio data. The data fusion module is used to fuse the first data and the second data to obtain the third data; A model building module is used to build a multimodal model, which includes at least an input layer, an intermediate layer and an output layer. The input layer includes a pre-trained large language model and a pre-trained audio encoder. The output layer includes an audio decoder. The intermediate layer includes at least one cross-attention layer. The pre-trained large language model is used to receive the first data and the third data, and encode them to obtain a first code corresponding to the first data and a second code corresponding to the third data; the pre-trained audio encoder is used to receive the audio data and encode it to obtain an audio residual vector quantization code; the cross-attention layer is used to perform cross-attention calculation using the second code as a query vector, the first code as a key vector or value vector, and the audio residual vector quantization code as a value vector or key vector to obtain a weighted audio feature vector corresponding to the audio residual vector quantization code as a key vector or value vector; the output layer is used to receive the weighted audio feature vector and generate audio. The model training module is used to train the multimodal model based on the audio data, the first data, and the third data to obtain the audio generation model.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, the memory storing a policy model, wherein when the computer program is executed by the processor, it implements the training method of the audio generation model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the training method for the audio generation model as described in any one of claims 1-7.

Images