Audio generation model training method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of machine learning, and discloses an audio generation model training method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring to-be-trained data, wherein the to-be-trained data comprises audio data, first data and second data; fusing the first data and the second data to obtain third data; an input layer of the multi-modal model is used for receiving audio data, first data and third data and outputting an audio residual vector quantization code, a first code corresponding to the first data and a second code corresponding to the third data, and a cross attention layer of the multi-modal model is used for fusing the first code, the second code and the audio residual vector quantization code; and obtaining a weighted audio feature vector. And training the multi-modal model to obtain an audio generation model. Therefore, the model can learn semantic and acoustic features of long texts and contexts, so that audio output which is more natural and more melodious and more conforms to inherent semantic and auditory perception features is generated.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a training method, apparatus, device and storage medium for an audio generation model. Background Art

[0002] Speech generation technology is being widely used in short videos, robots, audio reading, map navigation, live broadcast commentary and other scenarios. Currently, in customer service services in the financial field such as banking, insurance, and securities, AI agents widely use speech generation technology to generate conversational voices to complete tasks such as question-and-answer interactions, notification reminders, and marketing recommendations.

[0003] In the process of speech generation, related technologies usually rely on the volume, timbre and other characteristics of audio data as well as emotional characteristics to generate speech, but lack the integration of text semantics and context dependence, especially the processing and speech generation capabilities of long texts are weak, resulting in a bland rhythm of the generated speech and a poor user experience. Summary of the invention

[0004] The present application discloses a training method, device, computer and storage medium for an audio generation model, which solves the problem that the existing speech generation technology is insufficient in combining text semantics and context dependency, and the generated speech has a bland rhythm.

[0005] In a first aspect, the present application provides a method for training an audio generation model, comprising:

[0006] Acquire data to be trained, the data to be trained comprising audio data, first data, and second data, the first data being speech transcription text data and music sound effect data corresponding to the audio data, and the second data being description text data corresponding to the audio data;

[0007] Merging the first data and the second data to obtain third data;

[0008] Construct a multimodal model, the multimodal model at least comprising an input layer, an intermediate layer and an output layer, the input layer comprising a pre-trained large language model and a pre-trained audio encoder, the output layer comprising an audio decoder, and the intermediate layer comprising at least one cross-attention layer; the pre-trained large language model is used to receive the first data and the third data, and encode to obtain a first code corresponding to the first data and a second code corresponding to the third data; the pre-trained audio encoder is used to receive the audio data, and encode to obtain an audio residual vector quantization code; the cross-attention layer is used to perform cross-attention calculation using the second code as a query vector, the first code as a key vector or a value vector, and the audio residual vector quantization code as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization code as a key vector or a value vector; the output layer is used to receive the weighted audio feature vector and generate audio;

[0009] The multimodal model is trained according to the audio data, the first data and the third data to obtain an audio generation model.

[0010] In a second aspect, the present application provides an audio generation device, comprising:

[0011] A data acquisition module, used to acquire data to be trained, wherein the data to be trained includes audio data, first data, and second data, wherein the first data is speech transcription text data and music sound effect data corresponding to the audio data, and the second data is description text data corresponding to the audio data;

[0012] A data fusion module, used for fusing the first data and the second data to obtain third data;

[0013] A model construction module is used to construct a multimodal model, wherein the multimodal model includes at least an input layer, an intermediate layer and an output layer, wherein the input layer includes a pre-trained large language model and a pre-trained audio encoder, the output layer includes an audio decoder, and the intermediate layer includes at least one cross-attention layer; the pre-trained large language model is used to receive the first data and the third data, and encode to obtain a first code corresponding to the first data and a second code corresponding to the third data; the pre-trained audio encoder is used to receive the audio data and encode to obtain an audio residual vector quantization code; the cross-attention layer is used to perform cross-attention calculation using the second code as a query vector, the first code as a key vector or a value vector, and the audio residual vector quantization code as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization code as a key vector or a value vector; the output layer is used to receive the weighted audio feature vector and generate audio;

[0014] The model training module is used to train the multimodal model according to the audio data, the first data and the third data to obtain an audio generation model.

[0015] In a third aspect, the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the memory stores a policy model, wherein when the computer program is executed by the processor, it implements a method provided in any embodiment of the present application.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor implements a method as provided in any embodiment of the present application.

[0017] The training method, device, computer and storage medium of the above-mentioned audio generation model, the method comprises: obtaining data to be trained, the data to be trained comprises audio data, first data and second data; fusing the first data and second data to obtain third data; constructing a multimodal model with a new architecture; the output layer is used to receive weighted audio feature vectors to generate audio; training the multimodal model according to the audio data, first data and third data to obtain an audio generation model. On the one hand, due to the use of the descriptive text data corresponding to the audio data, the model can learn the specific pattern of the psychoacoustic features of the second data in the audio in addition to learning the basic audio features of the first data, thereby generating audio that is more natural and more in line with human auditory perception; on the other hand, the audio generation model integrates the ability of extracting semantic features of long texts and contexts based on a pre-trained large language model and the ability of extracting acoustic features and other features based on a pre-trained audio encoder through a cross-attention mechanism, thereby improving the recognition ability of the existing audio generation model for the semantic features of long texts and contexts, thereby generating audio that is more natural and more in line with semantic characteristics.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a schematic flow chart of the steps of a method for training an audio generation model provided in one embodiment of the present application;

[0021] Figure 2 It is a schematic diagram of the framework structure of a multimodal model provided in an embodiment of the present application;

[0022] Figure 3 It is a schematic flow chart of the steps of a residual vector quantization encoding method provided in an embodiment of the present application;

[0023] Figure 4 It is a schematic flow chart of the steps of a multimodal model training method provided in an embodiment of the present application;

[0024] Figure 5 is a structural schematic diagram of an audio generating device provided in an embodiment of the present application;

[0025] Figure 6 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0026] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0029] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first data and the second data are only used to distinguish different data, and their sequence is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0031] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0032] To facilitate understanding of the embodiments of the present application, some words involved in the embodiments of the present application are briefly explained below.

[0033] 1. Speech transcription text data: Speech transcription text data is data that transcribes the spoken information in the audio into text form.

[0034] 2. Music and sound effect data: a record of music or sound effect audio that can be read, stored, edited, processed, played or generated by a computer. Common forms include MIDI (Musical Instrument Digital Interface) and MusicXML.

[0035] 3. Descriptive text data: Feature data extracted or annotated from audio data may involve one or more acoustic features, including loudness, pitch (tone), timbre, reverberation, nonlinearity, masking effect, binaural effect, emotional features, and phoneme features.

[0036] 4. Large Language Model (LLM): Also known as a large language model, it refers to a computer model that has been trained with a large amount of text data and has advanced text understanding and generation capabilities. These models can be used for various language tasks, including text summarization, translation, question answering, etc.

[0037] 5. Transformer model: A neural network model in the field of natural language processing (NLP) that performs well in tasks such as machine translation, text generation, sentiment analysis, and topic classification.

[0038] The basic architecture of the Transformer model consists of an encoder and a decoder, which processes input data in parallel through a multi-head self-attention mechanism, allowing the model to simultaneously focus on all positions in the sequence during encoding and decoding, thereby capturing long-distance dependencies.

[0039] 6. T5 Large Language Model: A large language model based on the Transformer architecture that can handle a variety of text conversion tasks. T5 stands for "Text-to-Text Transfer Transformer", which is a text-to-text converter that learns rich language representations through the pre-training phase, allowing the model to adapt to specific tasks during the fine-tuning phase and achieve excellent performance on multiple downstream tasks. T5's architecture allows the same model, loss function, and hyperparameters to be applied to any NLP task, such as machine translation, document summarization, question answering, and classification tasks such as sentiment analysis.

[0040] 7.GPT-NeoX-20B: A pre-trained large language model based on the Decoder-Only architecture, whose weights are open source.

[0041] 8. SoundStream: An end-to-end neural audio encoder that uses deep learning techniques to compress and reconstruct audio signals. The core idea of ​​SoundStream is to use a neural network model to learn efficient audio representation, thereby achieving high-quality audio compression.

[0042] 9.EmoNet: An AI model specifically designed for audio emotion feature extraction, usually based on deep learning techniques such as convolutional neural networks (CNN), recurrent neural networks (RNN), or Transformer.

[0043] 10.Decoder-Only architecture: a neural network design pattern that contains only one Decoder component and no independent Encoder.

[0044] 11. Mel Frequency Cepstral Coefficients (MFCCs): An audio feature extraction method or audio feature form based on the human auditory system, widely used in speech recognition, audio processing and music information retrieval. Mel frequency cepstral coefficients are obtained by a series of processing steps on the audio signal, including pre-emphasis, framing, Fourier transform, Mel filter bank, discrete cosine transform, etc.

[0045] 12. Cross-Attention: A variant of self-attention that allows the model to focus on relevant features of another modality (such as images) while processing one modality (such as text). This mechanism is particularly important in multimodal tasks because it can enhance the information interaction between different modalities.

[0046] 13. MLM (Masked Language Model): MLM is a training task that focuses on learning language representation at the word and phrase level. The core idea of ​​MLM is to randomly replace some words (called tokens) in the input text with special [MASK] tags, and then let the model predict these masked words, thereby training the model's sensitivity to context.

[0047] For example, in the pre-training process of a neural network model, MLM is implemented through the following steps:

[0048] Data preparation: Randomly select sentences from a large text corpus and replace a certain proportion of the words with [MASK] tags.

[0049] Model input: Sentences containing the [MASK] tag are input into the language model, and the model needs to predict these masked words based on the context.

[0050] Loss calculation: Calculate the difference between the word predicted by the model and the actual word, usually using the cross entropy loss function.

[0051] Back propagation: Based on the results of the loss function, the parameters of the model are adjusted through the back propagation algorithm to minimize the prediction error.

[0052] Iterative training: Repeat the above process until the model's performance on the MLM task reaches a satisfactory level.

[0053] 14. NSP (Next Sentence Prediction): NSP is a training task that focuses on learning coherence and logical relationships at the sentence and paragraph levels. This task aims to train the model to predict whether two given sentences appear sequentially, that is, whether the second sentence is a natural continuation of the first sentence.

[0054] NSP encourages the model to learn the logical and semantic relationships between sentences, so that the model can better capture the connections between texts when dealing with downstream tasks that require understanding text coherence, such as question answering and natural language reasoning. For example, during the pre-training process of the T5 model, NSP is implemented through the following steps:

[0055] Data preparation: Randomly select sentence pairs from a large amount of text. If the second sentence is a direct continuation of the first sentence, it constitutes a positive example; if the second sentence has no direct coherence with the first sentence, it constitutes a negative example.

[0056] Model Input: Taking a sentence pair as input, the T5 model receives two sentences and processes them through its encoder layer.

[0057] Loss calculation: The model predicts whether a given sentence pair belongs to a “positive example” or a “negative example”, and uses the cross-entropy loss function to measure the difference between the model prediction and the actual label.

[0058] Back propagation: Based on the results of the loss function, the parameters of the model are adjusted through the back propagation algorithm to minimize the prediction error.

[0059] Iterative training: Repeat the above process until the model's performance on the NSP task reaches a satisfactory level.

[0060] 15. Residual Vector Quantization (RVQ): A technique for data compression. Especially when processing large data sets, RVQ can approximate the quantization error of high-dimensional vectors through additional quantizers, and gradually refine the quantization result through multiple low-complexity quantizers instead of relying on a high-complexity quantizer, while generating efficient coding and maintaining low storage and computing requirements.

[0061] 16. Generalized Residual Vector Quantization (GRVQ): GRVQ is an improvement on RVQ. Its core idea is to iteratively select a codebook and optimize it, and then re-quantize the data set to obtain a new residual vector for use in the next iteration.

[0062] 17. Codebook: A codebook is a set of predefined discrete vectors, or a set of codewords, used to approximate the original continuous feature vectors during quantization. A good codebook design should ensure that the distance between codewords is as large as possible to reduce quantization errors, and should be optimized based on the statistical characteristics of the audio data.

[0063] 18. Audio encoder: In a multimodal model, the audio encoder can be considered as a component that converts speech information into audio feature representations. The audio feature representations may include Mel-spectrograms, linear spectrum pairs (LSPs), Mel-frequency cepstral coefficients (MFCCs), etc., which can capture the acoustic characteristics of speech, such as pitch, timbre, speaking rate, etc.

[0064] 19. Tokenizer: Tokenizer is an important tool in text processing. It is responsible for cutting the input continuous text string into smaller tokens, which can be words, subwords, characters or symbols. The design of the tokenizer depends on the specific language, model requirements and application scenarios, and aims to improve the efficiency and accuracy of subsequent text processing tasks (such as text classification, sentiment analysis, TTS, etc.). In some embodiments, the tokenizer can integrate components such as chunkers and lemma encoders to realize all preprocessing functions such as chunking, word segmentation, lemma and encoding before the text input large model.

[0065] 20. Prompt Engineering: Also known as prompt word enhancement or prompt word engineering, prompt enhancement refers to the technology of guiding the model to perform specific tasks through carefully designed text prompts. These prompts can include instructions, context, examples and other elements, with the aim of enabling the model to better understand the task requirements and produce high-quality answers or generated text. The key to prompt enhancement is to design effective prompts so that the model can correctly understand the task and give appropriate responses.

[0066] 21. Chain-of-Thought (CoT) Prompting: Also known as chain-of-thought prompting, it is a technique that uses natural language prompts to guide the model to generate intermediate thinking steps to solve problems. This technique helps the model demonstrate its ability to solve complex problems by adding a series of step-by-step reasoning steps to the prompts, improves the transparency and interpretability of the model output, and the generalization ability of the model, and helps to find problems and correct errors by reviewing the steps.

[0067] 22. Instruction and reasoning task datasets: These are datasets used for instruction reinforcement and reasoning reinforcement, which can enhance the model's responsiveness to instructions and reasoning capabilities, including Muffin3, T0-SF, and datasets containing Chain of Thought (CoT) annotations.

[0068] 23.Adam (Adaptive Moment Estimation): A gradient-based optimization algorithm for first-order optimization problems of stochastic objective functions. The ADAM algorithm uses adaptive estimates of the first and second moments of the gradient to provide personalized learning rates for different parameters. The method is easy to implement, computationally efficient, has low memory requirements, and is particularly suitable for problems with large amounts of data or high parameter dimensions.

[0069] 24.AdamW (Adam Weight Decay Fix): An optimization algorithm widely used in deep learning, it is an improved version of the Adam optimizer. The main contribution of AdamW lies in its handling of weight decay. In the traditional Adam optimizer, weight decay is added to the loss function as a regularization term, while in AdamW, weight decay is directly applied to the parameter update step, which can more effectively control the complexity of the model and help improve the generalization ability of the model.

[0070] 25. Retrieval-Augmented Generation (RAG): An AI technology that combines retrieval and generation. The basic principle is to first use a retrieval system to find the most relevant information to the user's query from a large amount of data. This information is then used as context for the generative model, helping the model generate more accurate and targeted responses.

[0071] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0072] In the field of audio generation, especially speech generation technology, existing machine learning methods usually focus on extracting the physical acoustic features of audio, such as Mel-frequency cepstral coefficients, fundamental frequency (F0), the state of hidden Markov models, etc. These features help machine learning models learn the physical acoustic features of audio.

[0073] However, compared with existing acoustic sensors such as microphones, human auditory organs have the characteristics of nonlinearity, difficulty in quantification, physiological, and complexity, which makes the human auditory system more delicate and complex when processing sound. Therefore, the human auditory system does not directly perceive these physical acoustic features, but perceives more complex psychoacoustic features, such as loudness, pitch, timbre, masking effect, reverberation, binaural effect, etc. In addition, the psychoacoustic features of speech are often integrated with the semantic features and specific structures of language, so the speech generated by simply combining phoneme features and emotional features is not natural enough. Incorporating physical properties, psychoacoustic features and semantic features into the training of audio generation models can enable the model to learn audio features that are more in line with human auditory habits. At the same time, by integrating the long text and context processing capabilities of the large language model, the model can learn the internal pattern structure and semantic connotation of the language, thereby generating more natural, more pleasant, and more in line with the internal semantics and auditory perception characteristics of audio output.

[0074] In addition, the diffusion model used in the existing multimodal model has certain value in the field of audio generation, but compared with the large language model based on the Decoder-Only architecture, the diffusion model has disadvantages in consistency, controllability and response speed of longer speech. In the voice interaction scenario of financial customer service, customer service needs to adopt different speech techniques under the guidance of professional and rigorous norms, and solve problems for customers in the conversation as efficiently and quickly as possible. If the consistency and controllability are insufficient, the customer service's response may cause negative business and legal consequences. If the response speed is not fast enough, customers will lose patience, become disappointed with the AI ​​agent service, and directly choose manual service.

[0075] Compared with the diffusion model, the advantage of using a large language model based on the Decoder-Only architecture is that it makes it possible to improve the rule fidelity, instruction fidelity and controllability of the audio generation model based on RAG technology, and it can make full use of its own data and connect the existing business and R&D process systems. Among them, the own data can include speech libraries, regulations, music library, sound effect material library, etc., to further enhance the audio generation model's ability to follow instructions and the quality and compliance of the generated audio.

[0076] To solve the above problems, the present application embodiment proposes a training method for an audio generation model. Figure 1 , Figure 1 This is a flowchart illustrating the steps of a method for training an audio generation model provided in one embodiment of the present application.

[0077] like Figure 1 As shown, the training method of the audio generation model specifically includes steps S101 to S104.

[0078] S101. Obtain data to be trained, where the data to be trained includes audio data, first data, and second data. The first data is speech transcription text data and music sound effect data corresponding to the audio data, and the second data is description text data corresponding to the audio data.

[0079] Exemplarily, when it is necessary to train an audio generation model for a voice interaction task, the audio data in the training data may be voice data, the first data is voice transcription text data corresponding to the voice data, and the second data is descriptive text data of the features of the voice data. The features described by the second data may include psychoacoustic features such as loudness, pitch (tone), timbre, reverberation, nonlinearity, masking effect, binaural effect, etc., and may also include relatively objective features such as speaker gender, speaker role, style, habits, accent, emotion, pun, connotation, essence, dialect, etc. The audio data features provided by the descriptive text are a useful supplement to the physical acoustic features extracted by the machine.

[0080] Exemplarily, when it is necessary to train an audio generation model for a music sound effect generation task, the audio data in the training data is music or sound effect audio, and the first data is the music sound effect data corresponding to the music or sound effect audio. The music sound effect data can be a digital score in MIDI format or MusicXML format. These formats can better record the pitch, rhythm, dynamics and other information of the music, and provide a structured representation of the music for the model. The second data is text data describing the characteristics of the music data. These descriptions may include descriptions of the music style, emotion, instrument type, etc., as well as possible music expression instructions, such as speed, dynamics, pitch change, etc. These descriptive text data help the model learn the emotional and style characteristics of the music, so that it can better capture the expression intention and emotional color of the music when generating music sound effects.

[0081] S102: Merge the first data and the second data to obtain third data.

[0082] Specifically, the fusion method may be to combine the second data to perform instruction enhancement or reasoning enhancement on the first data, or to perform splicing.

[0083] For example, taking instruction enhancement as an example and constructing a financial customer service dialogue data set as an example, the first data includes:

[0084] Ask: "What is my account balance?"

[0085] Answer: "Your account balance is $X."

[0086] The second data includes:

[0087] The timbre of the answer is pleasant, with a steady and healthy pitch.

[0088] The third data obtained by fusion includes:

[0089] Instructions: "Please answer customer account balance inquiries in the form of 'Your account balance is $X' in a pleasant, steady, healthy voice."

[0090] Ask: "What is my account balance?"

[0091] Answer: "Your account balance is $X."

[0092] S103. Construct a multimodal model.

[0093] like Figure 2 As shown, Figure 2It is a schematic diagram of the framework structure of a multimodal model provided in an embodiment of the present application, and the multimodal model is used to execute the training method of the audio generation model. Among them, the audio generation device can be configured in a terminal or a server. The multimodal model includes at least an input layer, an intermediate layer and an output layer, the input layer includes a pre-trained large language model and a pre-trained audio encoder, the output layer includes an audio decoder, and the intermediate layer includes at least one cross-attention layer; the pre-trained large language model is used to receive the first data and the third data, and encode to obtain the first code corresponding to the first data and the second code corresponding to the third data; the pre-trained audio encoder is used to receive the audio data, and encode to obtain the audio residual vector quantization code; the cross-attention layer is used to use the second code as the query vector, the first code as the key vector or value vector, and the audio residual vector quantization code as the value vector or key vector for cross-attention calculation, and obtain the audio residual vector quantization code as the key vector or value vector corresponding to the weighted audio feature vector; the output layer is used to receive the weighted audio feature vector and generate audio.

[0094] The pre-trained large language model is used to receive the first data and the third data, and encode them to obtain a first code corresponding to the first data and a second code corresponding to the third data.

[0095] The large language model has good text processing and encoding capabilities. It can effectively extract contextual dependencies and semantic features in long texts and long conversations, and output feature representations in the form of encoding.

[0096] Specifically, in order to convert the first data and the third data into the first code and the second code that can be used for multimodal model training, it is necessary to perform text preprocessing according to the functional integration of the text preprocessing part of the pre-trained large language model, such as chunking, word segmentation, and lemma, or use certain modules in the large language model alone to perform text preprocessing, or directly input the text into the large language model to obtain the first code and the second code.

[0097] Exemplarily, the first data and the third data are respectively input into a word segmenter to obtain the lemma information corresponding to the first data and the third data. If GPT-NeoX-20B based on the Decoder-Only architecture is used as a pre-trained large language model of the input layer of the multimodal model, the input text is encoded, specifically including the following steps:

[0098] Step 1: When the text length does not exceed the model limit and does not require chunking, use a lemma encoder, such as AutoTokenizer.from_pretrained('EleutherAI / gpt-neox-20b'), to receive the input text and obtain lemma information.

[0099] Step 2: Use the GPT-NeoX-20B model, such as AutoModelForCausalLM.from_pretrained('EleutherAI / gpt-neox-20b'), to receive the lemma information and obtain the semantic representation.

[0100] For the GPT-NeoX-20B model, the last token of the hidden state H(T) of the last layer of the GPT-NeoX-20B model can be selected as the semantic representation E(T) of the entire text, and its expression is as follows:

[0101] E(T)=H(T)[-1]

[0102] Exemplarily, the T5 large language model based on the Encoder-Decoder architecture can also be used as a pre-trained large language model of the input layer of the multimodal model to encode the input text, which specifically includes the following steps:

[0103] Step 1: Divide the input text into blocks using a sliding window to obtain block text.

[0104] Step 2: Use a tokenizer and encoder, such as T5Tokenizer.from_pretrained('t5-large'), to receive the segmented text and obtain the lemma information.

[0105] Step 3: Use the T5 large language model, such as T5Model.from_pretrained('t5-large'), to receive lemma information and obtain semantic representation.

[0106] It should be noted that for the T5 large language model, the first token of the hidden state H(T) of the last layer of the T5 large language model corresponding to each text data is usually selected as the semantic representation E(T) of the entire text, and its expression is as follows:

[0107] E(T)=H(T)[0]

[0108] It should be noted that in step S103, the semantic representation output by the pre-trained large language model of the input layer of the constructed multimodal large model can select a specific token according to the architecture of the model or the richness of the semantic features contained.

[0109] It should be noted that, in step S103, if a block operation is performed on the text data, an alignment block operation may be performed on the audio data to align the audio data used for training with the block-divided text data.

[0110] Exemplarily, in step S103, the pre-trained audio encoder is used to receive the audio data and encode to obtain audio residual vector quantization encoding, wherein SoundStream can be used as an audio encoding model of the pre-trained audio encoder to perform residual vector quantization encoding on the audio, specifically including the following steps:

[0111] Step 1: Input the audio data x into the encoding layer of the audio coding model to obtain the feature vector f(x) of the audio data, which is initialized to the residual q=f(x).

[0112] Step 2: Input the obtained feature vector f(x) into the quantizer layer, which maps the continuous feature vector to a discrete index space to obtain a set of discrete indexes q. The quantization process can be expressed as:

[0113] q=Q(f(x))=arg min c∈C ||f(x)-c||

[0114] where Q is the quantization function, ||·|| represents the Euclidean distance, and c is an element in the predefined codebook C.

[0115] Step 3: Use the pre-trained decoder D to reconstruct the audio features from the index q Subtract the reconstructed features from the current residual r Get the new residual r new :

[0116]

[0117] in

[0118] Step 4: Repeat steps 2 and 3 until the residual r t Less than the threshold ∈ or reaches the predetermined number of iterations >, the process can be expressed as:

[0119]

[0120] Step 5: The final residual r T Quantize again and save the residue as audio residual vector quantization encoding.

[0121] In some embodiments, the pre-trained audio encoder includes an audio residual vector quantization encoder, an emotion feature extractor, and a feature fusion unit. Figure 3 , Figure 3The present invention provides a flowchart of the steps of a residual vector quantization encoding method provided by an embodiment of the present invention. The SoundStream model is used as an audio encoding model of a pre-trained audio encoder, and the EmoNet model is used as an emotion feature extractor of a pre-trained audio encoder to extract audio emotion features, and a feature fusion device that uses splicing or attention mechanism to perform feature fusion is constructed, such as Figure 3 As shown, residual vector quantization encoding is performed on the audio, specifically including the following steps:

[0122] S103a, converting the audio data into a spectrogram or a Mel-frequency cepstrum coefficient to obtain preprocessed audio data x.

[0123] S103b, input the preprocessed audio data x into the audio coding model to obtain the basic feature f base (x), which is expressed as follows:

[0124] f base (x) = SoundStream(x)

[0125] S103c, input the pre-processed audio data x into the emotion feature extractor to obtain the emotion feature f emo (x), which is expressed as follows:

[0126] f emo (x)=EmoNet(x)

[0127] S103d. The basic features and the sentiment features are fused by concatenation or attention mechanism to obtain the fused feature representation, which is expressed as follows:

[0128] f fused (x) = Concat(f base (x),f emo (x)

[0129] S103e, input the fused feature representation into the audio coding model to generate a residual vector quantization code q, which is expressed as follows:

[0130] q=Q(f fused (x)

[0131] Where Q is the quantization function.

[0132] For example, Figure 2As shown, the multimodal model may include an input layer, an intermediate layer, and an output layer. The input layer may include the pre-trained audio encoder and the pre-trained large language model, the intermediate layer may include a cross-attention layer and a large language model based on a Decoder-Only architecture including the cross-attention layer, the intermediate layer may also include a cross-attention layer and a large language model based on a Decoder-Only architecture connected in series with the cross-attention layer, the output layer may include an audio decoder and a vocoder connected in series with the audio decoder, the vocoder may be a part of the audio decoder, or may not be included in the multimodal large model.

[0133] Among them, the input layer of the multimodal model is used to receive the data to be trained and encode it, and output the first code, the second code, and the audio residual vector quantization code to the middle layer.

[0134] The data to be trained received by the input layer of the multimodal model includes audio data, first data, and second data, wherein the first data is speech transcription text data and music sound effect data corresponding to the audio data, and the second data is description text data corresponding to the audio data.

[0135] In some embodiments, the input layer of the multimodal model receives the data to be trained and encodes it, and outputs the first code, the second code, and the audio residual vector quantization code to the middle layer, specifically including: fusing the first data and the second data to obtain the third data; using a pre-trained audio encoder to process the audio data to obtain an audio residual vector quantization code, using a pre-trained large language model to process the first data and the third data to obtain a first code corresponding to the first data and a second code corresponding to the third data; outputting the first code, the second code, and the audio residual vector quantization code to the middle layer.

[0136] In other embodiments, the input layer of the multimodal model receives the data to be trained and encodes it, and after performing feature preprocessing operations on the data of different modalities, outputs the first code, the second code, and the audio residual vector quantization code to the intermediate layer, specifically including: using a pre-trained audio encoder to process the audio data to obtain an audio residual vector quantization code, using a pre-trained large language model to process the first data and the third data to obtain a first code corresponding to the first data and a second code corresponding to the third data; performing feature preprocessing operations such as pooling, linearization, and data alignment on the first code, the second code, and the audio residual vector quantization code; outputting the first code, the second code, and the audio residual vector quantization code to the intermediate layer; the pre-trained audio encoder is used to process the audio data to obtain an audio residual vector quantization code; the pre-trained large language model is used to process the first data and the third data to obtain a first code and a second code.

[0137] In other embodiments, the input layer of the multimodal model receives the data to be trained and encodes it, and outputs the first code, the second code, and the audio residual vector quantization code to the intermediate layer, and the intermediate layer performs feature preprocessing on the first code, the second code, and the audio residual vector quantization code, and performs cross-attention calculation, specifically including: if the cross-attention layer of the intermediate layer is part of the large language model based on the Decoder-Only architecture, the large language model based on the Decoder-Only architecture can perform feature preprocessing operations such as pooling, linearization, and data alignment on the first code, the second code, and the audio residual vector quantization code, or other components of the intermediate layer can perform feature preprocessing operations such as pooling, linearization, and data alignment on the first code, the second code, and the audio residual vector quantization code, and the other components of the intermediate layer may include a linear layer or a linear projection layer.

[0138] Among them, the middle layer is used to fuse the audio residual vector quantization code with the first code and the second code to generate the desired audio feature expression. The cross attention layer is used to use the second code as the query vector, the cross attention layer uses the first code as the key vector or value vector, and the audio residual vector quantization code as the value vector or key vector to perform cross attention calculation and obtain the cross attention layer weight.

[0139] Exemplarily, the second code is used as the query vector Q, the first code is used as the key vector K, and the audio residual vector quantization code is used as the value vector V to perform cross attention calculation, specifically including the following steps:

[0140] Step 1: Use linear interpolation or pooling operation to align the length of the training data to achieve length alignment of the audio residual vector quantization coding with the first coding and the second coding to obtain the length-aligned training data.

[0141] Step 2: Use linear transformation or projection method to align the dimension of the length-aligned training data to obtain the dimension-aligned training data.

[0142] Step 3: Before the cross-attention calculation, the training data of different modalities can be preliminarily fused through splicing or weighted summation operations. Alternatively, the attention guidance method can be used to use the multi-head attention mechanism in advance to guide the correlation of the training data of different modalities to obtain the processed training data.

[0143] Step 4: Use the first code as the key vector K, the second code as the query vector Q, and the audio residual vector quantization code as the value vector V to perform cross attention calculation. The corresponding calculation formula is:

[0144] Calculate the attention score S,

[0145]

[0146] Calculate the attention weight A,

[0147] A=softmax(S)

[0148] Weighted sum,

[0149] O=AV

[0150] The weighted audio residual vector output O is obtained.

[0151] If the large language model based on the Decoder-Only architecture contained in the middle layer is connected in series with the cross-attention layer, the large language model based on the Decoder-Only architecture is used to receive the weighted audio residual vector of the cross-attention layer to generate the expected audio feature expression. The expected audio feature expression can be in the form of a spectrogram or Mel-frequency cepstral coefficients, or an implicit representation of the expected audio.

[0152] If the cross-attention layer included in the middle layer is part of a large language model based on the Decoder-Only architecture, the large language model based on the Decoder-Only architecture is used to receive the audio residual vector quantization encoding and the first encoding and the second encoding to generate the desired audio feature expression.

[0153] The output layer is used to receive the desired audio feature expression generated by the intermediate layer, generate audio expressions such as Mel-frequency cepstral coefficients, or reconstruct an audible audio waveform.

[0154] Exemplarily, the output layer includes an audio decoder and a vocoder, wherein the audio decoder is used to receive the implicit representation of the desired audio generated by the intermediate layer to obtain the Mel-frequency cepstral coefficients of the desired audio; the vocoder is used to receive the Mel-frequency cepstral coefficients of the desired audio to obtain an audible audio signal.

[0155] Exemplarily, the output layer includes an audio decoder and a vocoder, with the Decoder component of SoundStream as the pre-trained audio decoder. The audio decoder is used to receive the residual vector quantization encoding of the desired audio generated by the intermediate layer to obtain the Mel-frequency cepstral coefficients of the desired audio; the vocoder is used to receive the Mel-frequency cepstral coefficients of the desired audio to obtain an audible audio signal.

[0156] Exemplarily, the output layer includes only one audio decoder, which is used to receive the implicit representation of the desired audio generated by the intermediate layer to obtain Mel-frequency cepstral coefficients of the desired audio.

[0157] The audio decoder is used to receive the desired audio feature expression generated by the intermediate layer and generate Mel-frequency cepstral coefficients or other explicit feature expressions of the audio.

[0158] S104. Train a multimodal model according to the audio data, the first data, and the third data to obtain an audio generation model.

[0159] Specifically, the parameters of the pre-trained audio encoder and the pre-trained large language model of the input layer of the multimodal model can be frozen; the parameters of at least one layer of the audio decoder of the intermediate layer or the output layer can be frozen; the loss can be calculated according to the audio data, the first data and the third data input by the multimodal model using the NSP method and / or the MLM method; the unfrozen parameters of the multimodal model can be updated using the AdamW optimizer according to the loss; the two steps of calculating the loss according to the audio data, the first data and the third data input by the multimodal model using the NSP method and / or the MLM method, and updating the unfrozen parameters of the multimodal model using the AdamW optimizer according to the loss are repeated until the loss is less than a threshold or the repetition reaches a certain number of times, thereby obtaining an audio generation model.

[0160] In some embodiments, the training of the multimodal model according to the audio data, the first data and the third data specifically includes the following steps:

[0161] Freeze the parameters of the pre-trained audio encoder and the pre-trained large language model of the input layer of the multimodal model; if the intermediate layer does not include the large language model based on Decoder-Only, freeze the parameters of several layers of the audio decoder of the output layer; if the intermediate layer includes the large language model based on Decoder-Only, the parameters of several layers of the intermediate layer and the audio decoder can be frozen, or all parameters of the audio decoder can be frozen to prevent overfitting; use the NSP method and / or the MLM method to calculate the loss according to the audio data, the first data and the third data input by the multimodal model, and the calculated loss may include the cross entropy loss; update the unfrozen parameters of the multimodal model using the AdamW optimizer according to the loss; repeat the two steps of using the NSP method and / or the MLM method to calculate the loss according to the audio data, the first data and the third data input by the multimodal model, and updating the unfrozen parameters of the multimodal model using the AdamW optimizer according to the loss, until the loss is less than a threshold or the repetition reaches a certain number of times, to obtain an audio generation model.

[0162] Among them, the multimodal model is trained using the MLM method to obtain an audio generation model, which specifically includes:

[0163] Masking operation: Randomly mask a certain proportion of text words (usually 15%) and their corresponding audio segments.

[0164] Model Input: The masked text and audio data are fed into the multimodal model.

[0165] Calculate the loss: y t It is the audio residual vector quantization encoding at time step t, calculates the cross entropy loss, and obtains the loss L, that is:

[0166] Where T is the sequence length

[0167] It is also possible to calculate the cross entropy loss between the predicted audio residual vector quantization coding and the actual audio residual vector quantization coding, calculate the loss between the predicted expected audio data and the actual input audio data, and weightedly combine the two losses to obtain the total loss.

[0168] Back propagation and parameter update: Based on the loss obtained, use an optimizer such as AdamW to update the model parameters.

[0169] Exemplarily, a pre-trained GPT-NeoX model is used as a large language model based on a Decoder-Only architecture in the middle layer, and audio residual vector quantization coding, first coding, and second coding are used as inputs of the middle layer to obtain the expected audio residual vector quantization coding output of the middle layer. The step of training the multimodal model according to the audio data, the first data, and the third data includes: freezing the parameters of the pre-trained audio encoder of the input layer and the pre-trained large language model of the input layer to prevent weakening of the feature extraction capability of the input layer neural network model for audio and text. Obtain a pre-trained GPT-NeoX model, freeze part of the layers of the pre-trained GPT-NeoX model to avoid overfitting; use the NSP method to train the multimodal model to obtain an audio generation model,

[0170] Among them, the NSP method is used to train the multimodal model, specifically including:

[0171] Data preparation: construct positive samples corresponding to the third data and the audio data and unrelated negative samples by randomly shuffling the pairing of the audio data and the third data.

[0172] Model input: Input positive samples and negative samples into the multimodal model to obtain predicted samples.

[0173] Calculate the loss: Calculate the cross entropy loss with the goal of maximizing the probability of correctly predicting positive and negative samples.

[0174] Back propagation and parameter update: Based on the loss obtained, use an optimizer such as AdamW to update the model parameters.

[0175] For example, see Figure 4 , Figure 4 This is a flowchart showing the steps of a multimodal model training method provided in an embodiment of the present application. The pre-trained GPT-NeoX model is used as a large language model based on the Decoder-Only architecture in the middle layer, and the audio residual vector quantization coding, the first coding, and the second coding are used as the input of the middle layer to obtain the expected audio residual vector quantization coding output of the middle layer, such as Figure 4 As shown, step S104 specifically includes:

[0176] S104a, freezing the parameters of the pre-trained audio encoder and the pre-trained large language model of the input layer to prevent weakening of the feature extraction capability of the input layer neural network model for audio and text.

[0177] S104b, freezing the parameters of at least one layer of the audio decoder of the intermediate layer or the output layer, may include obtaining a pre-trained GPT-NeoX model and freezing a portion of the layers of the pre-trained GPT-NeoX model to avoid overfitting.

[0178] S104c, integrating the NSP method and the MLM method to train the multimodal model, specifically including alternately or successively using the NSP method and the MLM method as described above according to the audio data, the first data and the third data input into the multimodal model and calculating the loss.

[0179] S104d. Update the unfrozen parameters of the multimodal model based on the AdamW optimizer according to the loss.

[0180] S104e, repeat steps S104c and S104d until the loss is less than a threshold or the repetitions reach a certain number of times, and obtain an audio generation model.

[0181] In some embodiments, the model training method further includes: obtaining generation prompt data, inputting the generation prompt data into the audio generation model, and obtaining audio corresponding to the generation prompt data, specifically including the following steps:

[0182] Step 1: Obtain generation prompt data and input it into the input layer of the multimodal model; if the generation prompt data is in audio form, input the generation prompt data in audio form into the audio encoder of the input layer of the multimodal model; if the generation prompt data is in text form, input the generation prompt data in text form into the pre-trained large language model of the input layer of the multimodal model.

[0183] Step 2: The multimodal model input layer receives the generation prompt data, generates audio residual vector quantization code, and outputs it to the multimodal model middle layer.

[0184] Step 3: The middle layer of the multimodal model receives the audio residual vector quantization encoding, generates the desired audio feature representation, and outputs it to the output layer.

[0185] Step 4: The multimodal model output layer receives the expected audio feature representation. If the audio decoder of the multimodal model output layer includes the function of a vocoder, or the multimodal model output layer includes a vocoder, then the audio corresponding to the generation prompt data is generated sound audio audible to the human ear; if the audio decoder of the multimodal model output layer does not include the vocoder function, and the multimodal model output layer does not include a vocoder, then the audio corresponding to the generation prompt data is a digital representation of the audio, and the digital representation includes a Mel-spectrogram, a linear spectrum pair (LSP), a Mel-cephalogram coefficient (MFCCs), etc.

[0186] In some embodiments, the model training method further includes: using the LoRA method to train the multimodal model or fine-tune the multimodal model according to the audio data, the first data and the third data to obtain an audio generation model.

[0187] Exemplarily, if in step S103, when constructing a large multimodal model, the pre-trained GPT-NeoX model is used as a large language model based on the Decoder-Only architecture in the middle layer, and the audio residual vector quantization coding, the first coding, and the second coding are used as the input of the middle layer to obtain the expected audio residual vector quantization coding output of the middle layer, then step S104 may specifically include: freezing the parameters of the pre-trained GPT-NeoX model and the output layer. Insert the LoRA module before several layers of the middle layer to initialize the LoRA weight matrix. Alternately use the NSP method and the MLM method to input the model and calculate the loss of the multimodal model. Calculate the gradient based on the loss, and use the optimizer (such as AdamW) to update the parameters of the LoRA layer. Obtain a multimodal model.

[0188] like Figure 5 As shown, Figure 5 1 is a schematic diagram of the structure of an audio generation device provided in an embodiment of the present application, wherein the audio generation device is used to execute the aforementioned training method of the audio generation model. The audio generation device can be configured in a terminal or a server.

[0189] like Figure 5 As shown, the audio generating device 100 includes a data acquisition module 101, a data fusion module 102, a model building module 103 and a model training module 104.

[0190] The data acquisition module 101 is used to acquire the data to be trained, the data to be trained includes audio data, first data and second data, the first data is the speech transcription text data and music sound effect data corresponding to the audio data, and the second data is the description text data corresponding to the audio data;

[0191] A data fusion module 102, used to fuse the first data and the second data to obtain third data;

[0192] The model construction module 103 is used to construct a multimodal model, which includes at least an input layer, an intermediate layer and an output layer. The input layer includes a pre-trained large language model and a pre-trained audio encoder, the output layer includes an audio decoder, and the intermediate layer includes at least one cross-attention layer; the pre-trained large language model is used to receive the first data and the third data, and encode to obtain a first code corresponding to the first data and a second code corresponding to the third data; the pre-trained audio encoder is used to receive the audio data and encode to obtain an audio residual vector quantization code; the cross-attention layer is used to perform cross-attention calculation using the second code as a query vector, the first code as a key vector or a value vector, and the audio residual vector quantization code as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization code as a key vector or a value vector; the output layer is used to receive the weighted audio feature vector and generate audio;

[0193] The model training module 104 is used to train the multimodal model according to the audio data, the first data and the third data to obtain an audio generation model.

[0194] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0195] The above-mentioned device can be implemented in the form of a computer program. Figure 6 Runs on the computer device shown.

[0196] See also Figure 6 , Figure 6 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server. Figure 6 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0197] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any training method for the audio generation model.

[0198] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0199] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any training method for the audio generation model.

[0200] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0201] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0202] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0203] Acquire data to be trained, the data to be trained comprising audio data, first data, and second data, the first data being speech transcription text data and music sound effect data corresponding to the audio data, and the second data being description text data corresponding to the audio data;

[0204] Merging the first data and the second data to obtain third data;

[0205] Construct a multimodal model, the multimodal model at least comprising an input layer, an intermediate layer and an output layer, the input layer comprising a pre-trained large language model and a pre-trained audio encoder, the output layer comprising an audio decoder, and the intermediate layer comprising at least one cross-attention layer; the pre-trained large language model is used to receive the first data and the third data, and encode to obtain a first code corresponding to the first data and a second code corresponding to the third data; the pre-trained audio encoder is used to receive the audio data, and encode to obtain an audio residual vector quantization code; the cross-attention layer is used to perform cross-attention calculation using the second code as a query vector, the first code as a key vector or a value vector, and the audio residual vector quantization code as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization code as a key vector or a value vector; the output layer is used to receive the weighted audio feature vector and generate audio;

[0206] The multimodal model is trained according to the audio data, the first data and the third data to obtain an audio generation model.

[0207] In some embodiments, before constructing the multimodal model, the processor is further used to implement: inputting the first data and the third data into a word segmenter respectively to obtain the lemma information corresponding to the first data and the third data. The pre-trained large language model is used to receive the first data and the third data, and encode to obtain a first code corresponding to the first data and a second code corresponding to the third data, and also includes: the pre-trained large language model is used to receive the lemma information corresponding to the first data and the third data, and encode to obtain a first code corresponding to the first data and a second code corresponding to the third data.

[0208] In some embodiments, the intermediate layer includes a cross-attention layer and a large language model based on a decoder-only architecture, and the output layer includes an audio decoder and a vocoder; the cross-attention layer is used to perform cross-attention calculation using the second code as a query vector, the first code as a key vector or a value vector, and the audio residual vector quantization code as a value vector or a key vector, and obtain a weighted audio residual vector quantization code corresponding to the audio residual vector quantization code as a key vector or a value vector; the large language model based on the decoder-only architecture is used to utilize the weighted audio residual vector quantization code and infer the expected audio residual vector quantization code; the audio decoder is used to receive the expected audio residual vector quantization code and decode it to obtain an expected audio feature representation, wherein the expected audio feature representation includes Mel-frequency cepstral coefficients; the vocoder is used to receive the expected audio feature representation and generate audio.

[0209] In some embodiments, when the processor is used for training the multimodal model according to the audio data, the first data and the third data, it is specifically used to implement:

[0210] Freeze the parameters of the pre-trained audio encoder and the pre-trained large language model of the input layer of the multimodal model;

[0211] Freezing parameters of at least one layer of an audio decoder of the intermediate layer or the output layer;

[0212] Using the NSP method and the MLM method to input the audio data, the first data and the third data according to the multimodal model and calculate the loss;

[0213] According to the loss, the unfrozen parameters of the multimodal model are updated based on the AdamW optimizer; the two steps of using the NSP method and the MLM method to input the audio data, the first data and the third data based on the multimodal model and calculating the loss and updating the unfrozen parameters of the multimodal model based on the AdamW optimizer according to the loss are repeated until the loss is less than a threshold or a certain number of repetitions are reached to obtain an audio generation model.

[0214] In some embodiments, the multimodal model includes at least an input layer, an intermediate layer and an output layer, the input layer includes a pre-trained large language model and a pre-trained audio encoder, the pre-trained audio encoder includes an audio residual vector quantization encoder, an emotion feature extractor and a feature fusion device; the audio residual vector quantization encoder is used to receive the audio data, and obtain basic features based on residual vector quantization or generalized residual vector quantization; the emotion feature extractor is used to receive the audio data, and perform emotion feature extraction to obtain emotion features; the feature fusion device fuses the basic features and the emotion features based on a splicing or attention mechanism method to obtain a fused feature representation; the audio residual vector quantization encoder is used to receive the fused feature representation, and obtain the audio residual vector quantization code based on residual vector quantization or generalized residual vector quantization.

[0215] In some embodiments, the processor is further used to implement: obtaining generation prompt data, inputting the generation prompt data into the audio generation model, and obtaining audio corresponding to the generation prompt data.

[0216] In some embodiments, the processor is further used to implement: using the LoRA method, training the multimodal model or fine-tuning the multimodal model according to the audio data, the first data and the third data to obtain an audio generation model.

[0217] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the audio generation model training methods provided in the embodiments of the present application.

[0218] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0219] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A training method for an audio generation model, comprising: Acquire data to be trained, the data to be trained comprising audio data, first data, and second data, the first data being speech transcription text data and music sound effect data corresponding to the audio data, and the second data being description text data corresponding to the audio data; Merging the first data and the second data to obtain third data; Constructing a multimodal model, the multimodal model comprising at least an input layer, an intermediate layer and an output layer, the input layer comprising a pre-trained large language model and a pre-trained audio encoder, the output layer comprising an audio decoder, and the intermediate layer comprising at least one cross-attention layer; The pre-trained large language model is used to receive the first data and the third data, and encode to obtain a first code corresponding to the first data and a second code corresponding to the third data; the pre-trained audio encoder is used to receive the audio data, and encode to obtain an audio residual vector quantization code; the cross-attention layer is used to perform cross-attention calculation using the second code as a query vector, the first code as a key vector or a value vector, and the audio residual vector quantization code as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization code as a key vector or a value vector; the output layer is used to receive the weighted audio feature vector and generate audio; The multimodal model is trained according to the audio data, the first data and the third data to obtain an audio generation model.

2. The method according to claim 1, characterized in that Before constructing the multimodal model, the method further includes: Input the first data and the third data into a word segmenter respectively, and obtain lemma information corresponding to the first data and the third data; The pre-trained large language model is used to receive the first data and the third data, and encode them to obtain a first code corresponding to the first data and a second code corresponding to the third data, and also includes: the pre-trained large language model is used to receive the lemma information corresponding to the first data and the third data, and encode them to obtain a first code corresponding to the first data and a second code corresponding to the third data.

3. The method according to claim 1, characterized in that: The middle layer of the multimodal model includes a cross-attention layer and a large language model based on a decoder-only architecture, and the output layer includes an audio decoder and a vocoder; Among them, the cross-attention layer is used to perform cross-attention calculation with the second code as the query vector, the first code as the key vector or value vector, and the audio residual vector quantization code as the value vector or key vector, and obtain the weighted audio residual vector quantization code corresponding to the audio residual vector quantization code as the key vector or value vector; the large language model based on the Decoder-Only architecture is used to utilize the weighted audio residual vector quantization code and infer the expected audio residual vector quantization code; the audio decoder is used to receive the expected audio residual vector quantization code and decode it to obtain the expected audio feature representation, and the expected audio feature representation includes Mel-frequency cepstral coefficients; the vocoder is used to receive the expected audio feature representation and generate audio.

4. The method according to claim 1, characterized in that: The training of the multimodal model according to the audio data, the first data and the third data comprises: Freeze the parameters of the pre-trained audio encoder and the pre-trained large language model of the input layer of the multimodal model; Freezing parameters of at least one layer of an audio decoder of the intermediate layer or the output layer; Calculating the loss according to the audio data, the first data and the third data input by the multimodal model using the NSP method and / or the MLM method; updating unfrozen parameters of the multimodal model using an AdamW optimizer according to the loss; Repeat the two steps of using the NSP method and / or the MLM method to input the audio data, the first data and the third data according to the multimodal model and calculate the loss, and using the AdamW optimizer to update the unfrozen parameters of the multimodal model according to the loss, until the loss is less than a threshold or the repetition reaches a certain number of times, to obtain an audio generation model.

5. The method according to claim 1, characterized in that The pre-trained audio encoder includes an audio residual vector quantization encoder, an emotional feature extractor and a feature fusion device; The audio residual vector quantization encoder is used to receive the audio data and obtain basic features using a residual vector quantization algorithm or a generalized residual vector quantization algorithm; The emotion feature extractor is used to receive the audio data and extract emotion features to obtain emotion features; The feature fuser fuses the basic features and the sentiment features based on a concatenation or attention mechanism to obtain a fused feature representation; The audio residual vector quantization encoder is used to receive the fused feature representation and obtain the audio residual vector quantization code using a residual vector quantization algorithm or a generalized residual vector quantization algorithm.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire generation prompt data, input the generation prompt data into the audio generation model, and obtain audio corresponding to the generation prompt data.

7. The method according to any one of claims 1 to 5, characterized in that: The step of training the multimodal model according to the audio data, the first data and the third data to obtain the audio generation model includes: Using the LoRA method, the multimodal model is trained or fine-tuned according to the audio data, the first data and the third data to obtain an audio generation model.

8. An audio generating device, characterized in that: include: A data acquisition module, used to acquire data to be trained, wherein the data to be trained includes audio data, first data, and second data, wherein the first data is speech transcription text data and music sound effect data corresponding to the audio data, and the second data is description text data corresponding to the audio data; A data fusion module, used for fusing the first data and the second data to obtain third data; A model construction module, used to construct a multimodal model, wherein the multimodal model comprises at least an input layer, an intermediate layer and an output layer, wherein the input layer comprises a pre-trained large language model and a pre-trained audio encoder, the output layer comprises an audio decoder, and the intermediate layer comprises at least one cross attention layer; The pre-trained large language model is used to receive the first data and the third data, and encode to obtain a first code corresponding to the first data and a second code corresponding to the third data; the pre-trained audio encoder is used to receive the audio data, and encode to obtain an audio residual vector quantization code; the cross-attention layer is used to perform cross-attention calculation using the second code as a query vector, the first code as a key vector or a value vector, and the audio residual vector quantization code as a value vector or a key vector, to obtain a weighted audio feature vector corresponding to the audio residual vector quantization code as a key vector or a value vector; the output layer is used to receive the weighted audio feature vector and generate audio; The model training module is used to train the multimodal model according to the audio data, the first data and the third data to obtain an audio generation model.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the memory stores a strategy model, wherein when the computer program is executed by the processor, the training method of the audio generation model as described in any one of claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the training method for an audio generation model as described in any one of claims 1 to 7.

Images