Audio generation method based on stream matching and multi-mode conditions

Abstract

The invention relates to an audio generation method based on stream matching and a multi-modal condition, and the method comprises the following steps: S1, collecting and labeling data, the data comprising two types: a text-audio pair and a text-video-audio pair; s2, encoding the audio, inputting the encoded audio into an audio converter, and decoding to obtain audio output; s3, the text description obtains a text code through a text encoder, the text code is fused into an audio converter through a layer-by-layer cross attention mechanism, and audio output is obtained through decoding; s4, enabling the video to pass through a video encoder to obtain the video feature of each frame, enabling the video to pass through a video converter, and enabling the feature output by each layer of the video converter to be fused with the audio converter; and S5, performing multi-stage training from single-mode condition control to multi-mode condition control. According to the method, the audio corresponding to the text description or the video can be accurately generated with high quality, and meanwhile, the audio and the video can be well aligned.

Description

Technical Field

[0001] The present invention relates to the technical field of audio generation, and in particular to an audio generation method based on stream matching and multimodal conditions. Background Art

[0002] The existing technologies include: (1) Text-to-Audio (T2A) generation technology, which mainly uses the generation method of the diffusion model and regards audio generation as a series of noise reduction processes; (2) Video-to-Audio (V2A) generation technology, V2A uses the T2A model to guide the audio generation of the T2A model by extracting semantic information from the video and estimating the timestamp of the audio in the video; (3) V2A generation technology (direct training), which directly extracts video features and trains the diffusion model from scratch.

[0003] These technologies are difficult to support multiple control conditions of different modalities at the same time, and it is difficult to meet the requirements for the quality of generated audio, especially the alignment with the video. Due to the lack of annotated data in the field of audio generation, audio generation under a single modality control condition cannot use enough audio data, which further affects the quality of the generated audio.

[0004] Therefore, it is necessary to provide an audio generation method based on stream matching and multimodal conditions, which can accurately and high-quality generate audio corresponding to text descriptions or videos, and at the same time, the audio and video can achieve a good alignment effect. Summary of the invention

[0005] The purpose of the present invention is to provide an audio generation method based on stream matching and multimodal conditions, which can accurately and high-quality generate audio corresponding to a text description or video, and at the same time, the audio and video can achieve a good alignment effect.

[0006] In order to solve the problems existing in the prior art, the present invention provides an audio generation method based on stream matching and multimodal conditions, comprising the following steps:

[0007] S1: Collect and annotate data. The data includes two categories: text-audio pairs and text-video-audio pairs.

[0008] S2: Encode the audio, input the encoded audio into the audio converter, and obtain the audio output after decoding;

[0009] S3: The text description is encoded by the text encoder, and the text encoding is fused into the audio converter by the layer-by-layer cross-attention mechanism, and the audio output is obtained after decoding;

[0010] S4: The video passes through the video encoder to obtain the video features of each frame, and then passes through the video converter. The features output by each layer of the video converter are integrated with the audio converter;

[0011] S5: Multi-stage training from unimodal to multimodal conditional control.

[0012] Optionally, in the audio generation method based on stream matching and multimodal conditions, the text in the data is manually annotated.

[0013] Optionally, in the audio generation method based on flow matching and multimodal conditions, the encoded audio, text and video are input into the audio converter and undergo multi-step Flow-Matching iterations of the audio converter.

[0014] Optionally, in the audio generation method based on stream matching and multimodal conditions, multi-stage training from single-modal to multimodal condition control includes the following steps:

[0015] Phase 1: Use text-audio data to train the text modality part, and obtain the T2A model after training;

[0016] Phase 2: Add text-video-audio data and text-audio data, train the text modality part and the video modality part at the same time, make the model use video as a conditional control, and obtain T2A and T&V2A models;

[0017] The third stage: add text-video-audio data and text-audio data for training. During training, part of the text-video-audio data is changed to video-audio data to obtain T2A, T&V2A, and V2A models.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] (1) The present invention adopts flow matching. Flow matching technology can accurately and efficiently generate images, videos, audio and other modalities, showing broad application prospects.

[0020] (2) The present invention can accurately and efficiently generate audio that meets the conditions by combining the control conditions of text, video and other modalities through stream matching technology. A single model can simultaneously support single text or video condition control (T2A, V2A), and multiple condition control (T&V2A).

[0021] (3) The present invention can accurately and high-quality generate audio corresponding to the text description or video, and the audio and video can achieve a good alignment effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flow chart of a method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0024] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0025] Existing technologies are difficult to support multiple control conditions of different modalities at the same time, and it is difficult to meet the requirements for the quality of generated audio, especially the alignment with the video. Limited by the lack of annotated data in the field of audio generation, audio generation under a single modality control condition cannot use enough audio data, which further affects the quality of the generated audio.

[0026] In order to solve the problems existing in the prior art, the present invention provides an audio generation method based on stream matching and multimodal conditions, such as Figure 1 As shown, the method comprises the following steps:

[0027] S1: Collect and annotate data. The data includes two categories: text-audio pairs and text-video-audio pairs.

[0028] Specifically, text-audio pairs: audio and corresponding text descriptions, where the text descriptions mainly come from manual annotations and the screening of existing text descriptions;

[0029] Text-video-audio pair: audio and video and the corresponding text description. The text description mainly comes from manual annotation and the screening of existing text descriptions. At the same time, it is necessary to screen whether the audio and video have basic consistency.

[0030] S2: Encode the audio, input the encoded audio into the audio converter (i.e., audio Transformer), and obtain the audio output after decoding;

[0031] S3: The text description is encoded by the text encoder, and the text encoding is fused into the audio converter by the layer-by-layer cross-attention mechanism, and the audio output is obtained after decoding;

[0032] S4: The video passes through the video encoder to obtain the video features of each frame, and then passes through the video converter. The features output by each layer of the video converter are integrated with the audio converter to guide the generation of audio and its alignment with the video frame;

[0033] Preferably, in S2-S4, the encoded audio, text and video are input into the audio converter and undergo multi-step Flow-Matching iterations of the audio converter to obtain high-quality generated audio.

[0034] S5: Continue to refer Figure 1 , the multi-stage training from single modality to multi-modality conditional control includes the following steps:

[0035] Phase 1: Training with text-audio data Figure 1 The text modality part in the T2A model obtained after training;

[0036] Phase 2: Join text-video-audio data and text-audio data and train simultaneously Figure 1 The text modality part and the video modality part in the model are used to control the video condition, and the T2A and T&V2A models are obtained.

[0037] During training, if the training data is text-audio data that does not contain video, the output of the video encoder is set to 0, otherwise it is real video encoding.

[0038] The third stage: add text-video-audio data and text-audio data for training. During training, discard the text description of some videos, thereby changing some text-video-audio data into video-audio data. Figure 1 The output of the Chinese text encoder is set to 0. After training, the T2A, T&V2A, and V2A models are obtained.

[0039] Multi-stage training makes full use of all modal data to train the main generation module audio transformer, and also uses the data of each module to train the unique conditional control modules of each module, such as the cross-attention module and the video transformer module. Multi-stage training also ensures the stability of cross-modal training.

[0040] In summary, compared with the prior art, the present invention has the following advantages:

[0041] (1) The present invention adopts flow matching. Flow matching technology can accurately and efficiently generate images, videos, audio and other modalities, showing broad application prospects.

[0042] (2) The present invention can accurately and efficiently generate audio that meets the conditions by combining the control conditions of text, video and other modalities through stream matching technology. A single model can simultaneously support single text or video condition control (T2A, V2A), and multiple condition control (T&V2A).

[0043] (3) The present invention can accurately and high-quality generate audio corresponding to the text description or video, and the audio and video can achieve a good alignment effect.

[0044] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. An audio generation method based on stream matching and multimodal conditions, characterized in that: The following steps are involved: S1: Collect and annotate data. The data includes two categories: text-audio pairs and text-video-audio pairs. S2: Encode the audio, input the encoded audio into the audio converter, and obtain the audio output after decoding; S3: The text description is encoded by the text encoder, and the text encoding is fused into the audio converter by the layer-by-layer cross-attention mechanism, and the audio output is obtained after decoding; S4: The video passes through the video encoder to obtain the video features of each frame, and then passes through the video converter. The features output by each layer of the video converter are integrated with the audio converter; S5: Multi-stage training from unimodal to multimodal conditional control.

2. The audio generation method based on stream matching and multimodal conditions as claimed in claim 1, characterized in that: Manually annotate the text in the data.

3. The audio generation method based on stream matching and multimodal conditions as claimed in claim 1, characterized in that: The encoded audio, text, and video are input into the audio converter and go through the multi-step Flow-Matching iteration of the audio converter.

4. The audio generation method based on stream matching and multimodal conditions as claimed in claim 1, characterized in that: The multi-stage training from unimodal to multimodal conditional control includes the following steps: Phase 1: Use text-audio data to train the text modality part, and obtain the T2A model after training; Phase 2: Add text-video-audio data and text-audio data, train the text modality part and the video modality part at the same time, make the model use video as a conditional control, and obtain T2A and T&V2A models; The third stage: add text-video-audio data and text-audio data for training. During training, part of the text-video-audio data is changed to video-audio data to obtain T2A, T&V2A, and V2A models.

Images