Method and system for recognizing behaviors in videos on basis of visual-semantic features

Abstract

The invention discloses a method for recognizing behaviors in videos on basis of visual-semantic features. The method comprises the following steps of: firstly extracting short-term spatial-temporal visual features by utilizing a three-dimensional convolutional neural network, so as to avoid the high calculation complexity caused by a light stream or intensive trajectory method; extracting semantic and spatial position information of a person and an object by utilizing a convolutional neural network-based object detector, constructing person-object spatial position features, fusing the person-object spatial position features with the spatial-temporal visual features, and improving the recognition correctness for interaction behaviors in a video by utilizing extra semantic information; andfinally, on the basis of the extracted short-term spatial-temporal visual features with universality, extracting specific long-term behavior features through a recurrent neural network so as to improve the behavior recognition correctness. The method is capable of solving the technical problem that existing video-oriented behavior recognition methods are high in calculation complexity and low in behavior recognition correctness and cannot extract long-term behavior features which run through the whole video time dimensionality.

Description

technical field [0001] The technical field of computer vision of the present invention, more specifically, relates to a method and system for behavior recognition in videos based on visual-semantic features. Background technique [0002] The problem of action recognition for video data types has become a popular research area in the field of computer vision. At present, there are three main methods for behavior recognition in videos: optical flow method, recurrent neural network method, and three-dimensional convolutional neural network method. [0003] For the optical flow method, the accuracy of behavior recognition is relatively high, but because of the high computational complexity of the optical flow method, it cannot achieve real-time calculation; the input data of the cyclic neural network mainly includes two types: one is to use The feature of a single frame image extracted by the convolutional neural network lacks time-domain correlation information, resulting in a...

AI Technical Summary

Problems solved by technology

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a method and system for behavior recognition in video based on visual-semantic features, the purpose of which is to solve the computational complexity existing in the existing behavior recognition method for video High, low accuracy of behavior recognition, and the inability to extract long-term behavioral features throughout the entire video time dimension

Images