Video dense event description method based on multi-mode heterogeneous feature fusion

Abstract

The invention relates to the field of computer vision, in particular to a video event description method based on multi-mode heterogeneous feature fusion. According to the method, an I3D convolutional network is used for editing a video and extracting dynamic visual features, and a VGGish model is used for extracting audio rhythm features; performing semantic representation on the scene object information, generating a scene map, obtaining an entity code, an attribute code and a relation code, and performing map embedding on the feature vector through map convolution; performing triple multi-mode cycle fusion on the three extracted features; adaptive multi-modal data balance enables dynamic vision and audio rhythm features to be mutually matched, and ensures reasonable event extraction; and the description reconstruction decoder is used for detecting the video event by utilizing a description reconstruction algorithm and generating description of the video scene event according to the pre-training language dictionary. The problem that meaningless descriptions such as co-occurrence descriptions are frequently generated in a video description method is effectively solved, and the relation of scene events is effectively mined by effectively utilizing multi-modal information.

Description

technical field [0001] The invention relates to computer vision and to the field of video description, in particular to a description method for video-intensive events based on fusion of multimodal heterogeneous features. Background technique [0002] The scope and scale of computer vision applications are currently the most extensive and common in artificial intelligence applications, and have already penetrated into many aspects of daily life and work, involving network security, system evaluation, monitoring, intelligent machines, etc. It plays an important role in promoting the development and progress of society. In the image recognition of computer vision artificial intelligence, it is mainly divided into static images and dynamic images. Static images mainly include pictures and other content, and dynamic images mainly include video and other content. For video description tasks, identifying the relationship between these events in the video and describing all events...

AI Technical Summary

Problems solved by technology

[0008] In order to effectively solve the disadvantages of frequent generation of co-occurrence descriptions and other meaningless descriptions in the video description method of long videos and dense events, and meet the requirements of multi-scene applications, the present invention provides a video dense video based on multi-modal heterogeneous feature fusion Event description method

Images