Audio generation method based on multi-modal large model

Abstract

The invention relates to an audio generation method based on a multi-modal large model, and the method comprises the following steps: S1, collecting and marking data, the data comprising two types: a text-audio pair and a video-audio-text pair; s2, training based on texts, videos and audios to obtain a multi-modal large model; s3, carrying out first-stage training: fixing parameters of a text encoder, a video encoder, an audio encoder and the multi-mode large model, inputting collected and labeled data into the multi-mode large model, and training an audio decoder by taking audio output as a target; and S4, performing second-stage training: fixing parameters of a text encoder, a video encoder and an audio encoder, inputting collected and labeled data into the multi-modal large model, and training the multi-modal large model and an audio decoder at the same time by taking audio and text output as a target. According to the method, the audio corresponding to the text description and the video can be accurately generated with high quality, and meanwhile, the audio and the video can be well aligned.

Description

Technical Field

[0001] The present invention relates to the technical field of audio generation, and in particular to an audio generation method based on a multimodal large model. Background Art

[0002] Audio generation is a very important technical direction in various application scenarios of generative artificial intelligence, such as background sound generation and video dubbing. People have increasingly higher requirements for the accuracy and quality of generated audio.

[0003] The existing technologies include: (1) Text-to-Audio (T2A) generation technology, which mainly uses the generation method of the diffusion model and regards audio generation as a series of noise reduction processes; (2) Video-to-Audio (V2A) generation technology, V2A uses the T2A model to guide the audio generation of the T2A model by extracting semantic information from the video and estimating the timestamp of the audio in the video; (3) V2A generation technology (direct training), which directly extracts video features and trains the diffusion model from scratch.

[0004] Existing technologies are limited by the amount of high-quality text, audio, and video annotation data, and the above solutions are difficult to guarantee the accuracy and high quality of generated audio. Lacking the support of large data volumes and large model capabilities, the accuracy of text and video understanding and the richness of generated audio are insufficient.

[0005] Therefore, it is necessary to provide an audio generation method based on a multimodal large model, which can accurately and high-quality generate audio corresponding to the text description and video, and at the same time, the audio and video can achieve a good alignment effect. Summary of the invention

[0006] The purpose of the present invention is to provide an audio generation method based on a multimodal large model, which can accurately and high-quality generate audio corresponding to text description and video, and at the same time, the audio and video can achieve a good alignment effect.

[0007] In order to solve the problems existing in the prior art, the present invention provides an audio generation method based on a multimodal large model, comprising the following steps:

[0008] S1: Collect and annotate data. The data includes two categories: text-audio pairs and video-audio-text pairs.

[0009] S2: Get a large multimodal model based on text, video and audio training;

[0010] S3: Perform the first stage of training: fix the parameters of the text encoder, video encoder, audio encoder, and multimodal large model, input the collected and annotated data into the multimodal large model, and train the audio decoder with the goal of outputting audio;

[0011] S4: Perform the second stage of training: fix the parameters of the text encoder, video encoder, and audio encoder, input the collected and labeled data into the multimodal large model, aiming to output audio and text, and train the multimodal large model and audio decoder at the same time.

[0012] Optionally, in the audio generation method based on the multimodal large model, the text in the data is manually annotated.

[0013] Optionally, in the audio generation method based on the multimodal large model, the multimodal large model is trained in the following manner:

[0014] Text, video and / or audio are passed through encoders of different modalities.

[0015] Convert encoded text, video, and audio into input sequences;

[0016] The input sequence is input into the large model for training to obtain a multimodal large model.

[0017] Optionally, in the audio generation method based on the multimodal large model, when the collected and annotated data is input into the multimodal large model during the first stage training and the second stage training, the following steps are also included:

[0018] Text, video and / or audio need to go through encoders of different modalities;

[0019] Convert encoded text, video, and audio into input sequences.

[0020] Optionally, in the audio generation method based on the multimodal large model, the input video sequence frames and the output audio sequence frames correspond one-to-one to achieve audio and video alignment.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] The present invention utilizes multimodal large model technology to process different modal information (text, video, audio), and can accurately and high-quality generate audio corresponding to the text description and video, while the audio and video can achieve a good alignment effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of a method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0025] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0026] Existing technologies are limited by the amount of high-quality text, audio, and video annotation data, and the above solutions are difficult to guarantee the accuracy and high quality of generated audio. Lacking the support of large data volumes and large model capabilities, the accuracy of text and video understanding and the richness of generated audio are insufficient.

[0027] In order to solve the problems existing in the prior art, the present invention provides an audio generation method based on a multimodal large model, such as Figure 1 As shown, the following steps are included:

[0028] S1: Collect and annotate data. The data includes two categories: text-audio pairs and video-audio-text pairs.

[0029] Specifically, text-audio pairs: audio and corresponding text descriptions, where the text descriptions mainly come from manual annotations and the screening of existing text descriptions;

[0030] Video-audio-text pair: audio and video and corresponding text descriptions. The text descriptions mainly come from manual annotations and the screening of existing text descriptions. At the same time, it is necessary to screen whether the audio and video have basic consistency.

[0031] S2: A multimodal large model is obtained based on text, video and audio training as follows:

[0032] Text, video and / or audio are passed through encoders of different modalities.

[0033] Convert encoded text, video, and audio into input sequences;

[0034] like Figure 1 As shown, the input sequence is input into the large model for training, and the corresponding text output (generally a description or response to audio and video input) is generated to obtain a multimodal large model.

[0035] The large multimodal model is trained with massive multimodal data, and has particularly good capabilities in processing and understanding audio and video information, and can well assist T2A and V2A in generating audio.

[0036] S3: Perform the first stage of training: fix the parameters of the text encoder, video encoder, audio encoder, and multimodal large model, input the collected and labeled data into the multimodal large model, train the audio decoder with the output audio as the goal; this stage allows the audio decoder to initially have the ability to generate audio based on the output of the multimodal large model.

[0037] S4: Perform the second stage of training: fix the parameters of the text encoder, video encoder, and audio encoder, input the collected and annotated data into the multimodal large model, aim to output audio and text, and train the multimodal large model and audio decoder at the same time. In this stage, the audio decoder and the multimodal large model are jointly trained to further fine-tune the audio generation capability. At the same time, by training the text output capability of the multimodal large model, the large model's ability to understand multimodal information is retained.

[0038] Furthermore, during the first stage training and the second stage training, when the collected and annotated data is input into the multimodal large model, the following steps are also included:

[0039] Text, video and / or audio need to go through encoders of different modalities;

[0040] Convert encoded text, video, and audio into input sequences.

[0041] Preferably, the input video sequence frames and the output audio sequence frames correspond one to one to achieve audio and video alignment.

[0042] In summary, compared with the prior art, the present invention has the following advantages:

[0043] The present invention utilizes multimodal large model technology to process different modal information (text, video, audio), and can accurately and high-quality generate audio corresponding to the text description and video, while the audio and video can achieve a good alignment effect.

[0044] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. An audio generation method based on a multimodal large model, characterized in that: The following steps are involved: S1: Collect and annotate data. The data includes two categories: text-audio pairs and video-audio-text pairs. S2: Get a large multimodal model based on text, video and audio training; S3: Perform the first stage of training: fix the parameters of the text encoder, video encoder, audio encoder, and multimodal large model, input the collected and annotated data into the multimodal large model, and train the audio decoder with the goal of outputting audio; S4: Perform the second stage of training: fix the parameters of the text encoder, video encoder, and audio encoder, input the collected and labeled data into the multimodal large model, aiming to output audio and text, and train the multimodal large model and audio decoder at the same time.

2. The audio generation method based on a multimodal large model as claimed in claim 1, characterized in that: Manually annotate the text in the data.

3. The audio generation method based on a multimodal large model as claimed in claim 1, characterized in that: The multimodal large model is trained as follows: Text, video and / or audio are passed through encoders of different modalities. Convert encoded text, video, and audio into input sequences; The input sequence is input into the large model for training to obtain a multimodal large model.

4. The audio generation method based on a multimodal large model as claimed in claim 1, characterized in that: During the first and second phases of training, when the collected and annotated data is input into the multimodal large model, the following steps are also included: Text, video and / or audio need to go through encoders of different modalities; Convert encoded text, video, and audio into input sequences.

5. The audio generation method based on a multimodal large model as claimed in claim 4, characterized in that: The input video sequence frames and the output audio sequence frames correspond one to one to achieve audio and video alignment.

Images