Automatic audio summary generation method and device

Abstract

The invention discloses an automatic audio abstract generation method and device, and the method comprises the steps: pre-training a sound event detection model which comprises an audio feature extraction part and an output part; enabling the audio encoder to take the audio feature extraction part as an audio abstract automatic generation model; and training the audio abstract automatic generation model in an end-to-end manner. According to the scheme provided by the embodiment of the invention, a better audio encoder is obtained through pre-training and transfer learning on the sound event detection task, so that more accurate audio abstract description is generated, corresponding text description can be generated for any new audio, the audio-text database is automatically established, and practical application of similar audio retrieval engines based on natural languages in unlimited forms can be supported.

Description

technical field

[0001] The invention belongs to the technical field of audio summaries, in particular to a method and device for automatically generating audio summaries. Background technique

[0002] In related technologies, Automated audio captioning (Automated audio captioning, AAC) aims to generate a summary description of an audio clip. Many concepts are described in audio summarization, ranging from local information such as sound events to global information such as the acoustic scene. Currently, the mainstream method of AAC is an end-to-end encoder-decoder structure, and it is hoped that the encoder can automatically learn all the concepts embedded in the audio.

[0003] The automatic audio summary generation task can be based on an input audio, an encoder encodes the audio into a series of vectors, and then a decoder decodes the encoded vectors into natural language summaries. The inventor found in the process of implementing the present application that the gener...

Images