Audio-visual video question and answer method and system based on multi-modal heterogeneous graph

Abstract

The invention discloses an audiovisual video question and answer method and system based on a multi-modal heterogeneous graph, and relates to the technical fields of multi-modal interaction, video content analysis, intelligent question and answer and the like. According to the visual features and the audio features, constructing a motion-guided heterogeneous graph and an audio-guided heterogeneous graph respectively; aggregating the motion-guided and audio-guided heterogeneous graphs to obtain visual features and audio features after audio-visual interaction; extracting question features in the query question, selecting a key visual area according to the question features, and obtaining sparse visual features; according to the problem features and the sparse visual features, a problem-guided heterogeneous graph is constructed, and local visual features and global visual features are obtained; fusing the question features with the audio features, the local visual features and the global visual features after audio-visual interaction to obtain fused question features; and according to the question features after multi-modal fusion, a prediction result of the question answer is obtained, and the question and answer accuracy is improved.

Images