Video depth relation analysis method based on multi-modal feature fusion

Abstract

The invention relates to a video depth relation analysis method based on multi-modal feature fusion is based on a visual, sound and character feature fusion network of video sub-screens, scenes and character recognition; and the method comprises the following steps: firstly, dividing an input video into a plurality of screens according to scene, visual and sound models, and extracting corresponding sound and character features on each screen; secondly, identifying positions appearing in each screen according to the input scene screenshots and figure screenshots, extracting corresponding entityvisual features from the scene and the figure, and calculating visual features of a joint area for every two entity pairs; and for each entity pair, connecting the screen features, the entity features and the entity pair features, predicting a relationship between each screen entity pair through small sample learning in combination with zero sample learning, and constructing an entity relationship graph on the whole video by combining the entity relationship on each screen of the video. According to the method, three types of deep video analysis questions including knowledge graph filling, question answering and entity relationship paths can be answered by utilizing the entity relationship graph.

Description

technical field [0001] The invention belongs to the technical field of computer vision, relates to entity relationship detection in video, and specifically relates to a video depth relationship analysis method based on multimodal feature fusion. Background technique [0002] In-depth relationship analysis between different entities on long videos is helpful for in-depth understanding of long videos, which often requires inferring hidden information from known information. In-depth relationship analysis on long videos is dedicated to constructing relationship graphs between two types of entities, scenes and characters. Through the entity relationship diagram, various questions about in-depth video analysis can be answered. [0003] Similar work on video understanding includes video induction, behavior recognition, visual relationship detection, and social relationship recognition, but these works are generally applicable to short videos, and lack in-depth analysis of relatio...

AI Technical Summary

Problems solved by technology

[0004] 1) The content of the short video is relatively small, often there is only one scene, and there are not many characters. The existing technology for short video analysis cannot solve multiple entities, including the relationship prediction between characters and scenes;

[0005] 2) It is difficult to merge the analysis of short videos, and it is impossible to predict the relationship between entities that are not in the same frame

[0006] At the same time, most of the existing analysis methods are applicable to the situation where there are enough training samples, while the deep relationship analysis task of long video has some relationships without training samples.

Therefore, the existing technology cannot solve the deep relationship analysis on the long video

Images