A video description method and system based on an information loss function

Abstract

The invention relates to a video description method and system based on an information loss function, and the method comprises the steps: obtaining a training video, and obtaining the semantic information of each frame of a set training video; Inputting the semantic information of the training video into an LSTM-combined hierarchical attention mechanism model to obtain character description of thetraining video; According to the importance of each word in the character description to the expression video content, performing loss weighting on the words to obtain an information loss function, and taking the information loss function as an objective function to perform back-propagation gradient optimization on the hierarchical attention mechanism model to obtain a video description model; Obtaining a to-be-described video, respectively inputting the to-be-described video into the target detection network, the convolutional neural network and the action recognition network to obtain a setof target features, overall features and motion features of each frame of the to-be-described video as semantic information of the to-be-described video, and inputting the semantic information into the video description model to obtain character description of the to-be-described video.

Description

technical field [0001] The invention relates to the technical field of computer vision and natural language processing, in particular to a video description method and system based on an information loss function, which can be applied to video description, human-computer interaction and video retrieval tasks. Background technique [0002] The video describes the model architecture. Current video description models are mainly divided into bottom-up and top-down structures. The bottom-up model first recognizes limited semantic words based on visual information, and then connects these semantic words into a sentence through language templates. The study found that bottom-up generated statements lack flexibility. Inspired by machine translation tasks, researchers proposed a top-down model based on long short-term memory (hereinafter referred to as LSTM). The top-down model completes semantic word recognition while generating sentences, so this method can generate more diverse...

AI Technical Summary

Problems solved by technology

[0005] When the inventor was conducting visual description research, he found that the descriptions generated by existing video description methods had the problems of missing details and recognition errors

This problem is caused by the fact that the existing loss function is affected by the uneven distribution of words in the data set and the visual features used by the existing methods are not rich enough.

The problem of uneven distribution of words can be simply attributed to the problem of uneven samples. The model spends a lot of effort to learn a small number of simple samples, which leads to the recognition of discriminative words by the model.

However, directly adopting methods to solve sample unevenness (such as increasing the loss weight of rare words) will cause the model to pay too much attention to rare words

And rare words are not necessarily related to the video content, causing problems in model learning

Images