Method and system for recognizing behaviors in videos on basis of visual-semantic features

Abstract

The invention discloses a method for recognizing behaviors in videos on basis of visual-semantic features. The method comprises the following steps of: firstly extracting short-term spatial-temporal visual features by utilizing a three-dimensional convolutional neural network, so as to avoid the high calculation complexity caused by a light stream or intensive trajectory method; extracting semantic and spatial position information of a person and an object by utilizing a convolutional neural network-based object detector, constructing person-object spatial position features, fusing the person-object spatial position features with the spatial-temporal visual features, and improving the recognition correctness for interaction behaviors in a video by utilizing extra semantic information; andfinally, on the basis of the extracted short-term spatial-temporal visual features with universality, extracting specific long-term behavior features through a recurrent neural network so as to improve the behavior recognition correctness. The method is capable of solving the technical problem that existing video-oriented behavior recognition methods are high in calculation complexity and low in behavior recognition correctness and cannot extract long-term behavior features which run through the whole video time dimensionality.

Description

technical field [0001] The technical field of computer vision of the present invention, more specifically, relates to a method and system for behavior recognition in videos based on visual-semantic features. Background technique [0002] The problem of action recognition for video data types has become a popular research area in the field of computer vision. At present, there are three main methods for behavior recognition in videos: optical flow method, recurrent neural network method, and three-dimensional convolutional neural network method. [0003] For the optical flow method, the accuracy of behavior recognition is relatively high, but because of the high computational complexity of the optical flow method, it cannot achieve real-time calculation; the input data of the cyclic neural network mainly includes two types: one is to use The feature of a single frame image extracted by the convolutional neural network lacks time-domain correlation information, resulting in a...

AI Technical Summary

Problems solved by technology

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a method and system for behavior recognition in video based on visual-semantic features, the purpose of which is to solve the c

Images