Short video classification method based on multi-modal feature complete representation

Abstract

The invention discloses a short video classification method based on multi-modal feature complete representation, and the method comprises the steps: for the content information of a short video, providing constructing four subspaces from the perspective of modal missing mainly based on a visual modal feature, and obtaining potential feature representations respectively, further fusing the potential feature representations of the four subspaces by using an automatic coding and decoding network to ensure that more robust and effective public potential representations are learned; for label information, using inverse covariance estimation and a graph attention network to explore correlation between labels and update label representation to obtain label vector representation corresponding to the short video; providing a multi-head cross-modal fusion scheme based on multi-head attention for the public potential representation and the label vector representation, wherein the multi-head cross-modal fusion scheme is used for obtaining a label prediction score of the short video, wherein the overall loss function of the model is composed of traditional multi-label classification loss and reconstruction loss of an automatic coding and decoding network and is used for measuring the difference between a network output value and an actual value and guiding the network to find an optimal solution of the model.

Description

technical field

[0001] The invention relates to the field of short video classification, in particular to a short video classification method based on complete representation of multimodal features. Background technique

[0002] In recent years, with the popularity of smart terminals and the popularity of social networks, more and more information is presented in multimedia content. High-definition cameras, large-capacity storage and high-speed network connections have created extremely convenient shooting and sharing conditions for users, thus creating massive amounts of multimedia data.

[0003] As a new type of user-generated content, short videos have been greatly welcomed in social networks due to their unique advantages such as low barriers to creation, fragmented content, and strong social attributes. Especially since 2011, with the popularization of mobile Internet terminals, the speed-up of the network and the reduction of traffic charges, short videos have quickly...

Images