Video question answering method based on object-oriented double-flow attention network

Abstract

The invention discloses a video question answering method based on an object-oriented double-flow attention network. Visual content of a video is represented using a double stream mechanism, one stream being a static appearance stream of a foreground object and the other stream being a dynamic behavior stream of the foreground object. In each stream, the features of the object include the features of the object itself, and also include the space-time coding of the object and the context information features of the scene where the object is located. The relative space-time relation and the context sensing relation between the objects can be explored when deep feature extraction is carried out in subsequent image convolution operation. Meanwhile, a double-flow mechanism is used for solving the problem that a previous video question-answer model only considers static characteristics of an object and lacks dynamic information analysis. According to the method, the exploration capability on intra-modal interaction and inter-modal semantic alignment is improved, and a better result is obtained on a related video question and answer data set.

Description

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine

Login to View More

AI Technical Summary

Benefits of technology

This patented technology uses two different ways: one allows users to see what they are doing with videos while another analyzes how things behave when interacting within them. By analyzing these properties at once, it provides better understanding about interactions among multiple sources like backgrounds or scenes. It also helps improve upon existing models by providing more accurate representations of real world events such as human actions or movements. Overall, this method enhances learning from data collected through experiments conducted over large amounts of digital media.

Problems solved by technology

In order to solve problem addressed by these systems related to video query answers (QA), they require understanding how well things like backgrounds or other parts of the scene look up from frame data alone while still taking into account their relationships during QA analysis. However, current models have limitations when trying to capture complex behaviors involving multiple modes of interest.

Images