Double-stream video classification method and device based on cross-mode attention mechanism

Abstract

The invention relates to a double-stream video classification method and device based on a cross-modal attention mechanism. The method is different from a traditional double-flow method. Information of two modes (even more modes) is fused before a prediction result. Therefore, the method is more efficient and sufficient. Meanwhile, information interaction is carried out in the earlier stage, a single branch has important information of another branch in the later stage, the precision of the single branch is flush with or even exceeds that of a traditional double-flow method, and the parameterquantity of the single branch is much less than that of the traditional double-flow method. Compared with a non-local neural network, the attention module designed by the invention can be in a cross-mode state instead of only using an attention mechanism in a single mode, and the effect of the method provided by the invention is equivalent to that of the non-local neural network under the condition that the two modes are the same.

Description

technical field [0001] The invention relates to a video classification method, in particular to a dual-stream video classification method and device using an attention mechanism, belonging to the field of computer vision. [0002] technical background [0003] With the rapid development of deep learning in the image field, deep learning methods have gradually been introduced in the video field and have achieved certain achievements. However, the current technical level is far from reaching the desired effect, and the problems faced mainly include the following two aspects: [0004] First, current technology has yet to take full advantage of dynamic information. The difference between video and image is that the dynamic information between frames is unique and very important to video. For example, even for humans, it is difficult to judge various sub-categories of dances (such as tango and salsa) only by looking at one frame of images, and if the motion trajectory informatio...

AI Technical Summary

Problems solved by technology

[0005] Second, the current technology is still difficult to quickly and accurately locate key objects

As for how to capture key clues, that is, to introduce the attention mechanism into video classification, there are relatively few studies. The more representative one is the non-local neural network (Non-local Neural Networks), but the network can only focus on a single Important information inside the modal, there is no special way to model "moving objects"

Images