Video annotation method based on multiple modes

Abstract

The invention discloses a video annotation method based on multiple modes, and belongs to the technical field of computer vision and video annotation. The method comprises the following steps: extracting a key frame of a video through a clustering method; extracting features of the key frames, and aggregating continuous key frame features through a learning pool to generate visual features of thevideo; extracting audios in the video, and dividing the audios into a plurality of independent frames; extracting audio frame features, and aggregating the continuous audio frame features through a learning pool to generate audio features of the video; fusing the visual features and the audio features and inputting into a prediction module; and performing video tagging. Compared with the prior art, the method has the advantages that visual features and audio features of the video are considered at the same time, and an attention mechanism is added during frame feature aggregation, so that theextracted video features are more representative, and the video annotation accuracy is greatly improved.

Description

Technical field [0001] The present invention belongs to the technical field of computer vision and video annotation, and more specifically, relates to a video annotation method based on multi-modality. Background technique [0002] With the continuous development of the Internet, Internet portals with video applications as the theme have rapidly developed in my country, and video has become a way for people to communicate with each other and share their lives. Every day, a large number of videos are uploaded to domestic video sharing websites, such as Youku and Douyin. Compared with voice, text and other media files, the data structure of video is more complicated, and it provides more useful information, and its content is more vivid, vivid and intuitive. Although video data contains a wealth of information, which is unmatched by other data forms, its complex data format and its ever-increasing amount of data undoubtedly set up huge obstacles for user interaction and affect its...

AI Technical Summary

Problems solved by technology

At present, most machine learning methods only label videos based on visual features, but video files are more complex, which contains not only image information, but also audio information. It is not accurate enough to label videos only by visual features of videos, and currently The frame-level feature aggregation method does not take into account the importance of the frame to the video, which greatly reduces the quality of the aggregated features

Images