Action recognition method and device based on first person view angle

Abstract

The invention discloses an action recognition method and device based on a first person view angle. The method comprises the following steps: acquiring a to-be-processed RGB video frame, wherein the to-be-processed RGB video frame comprises hand action image information based on a first person view angle; inputting all the RGB video frames to be processed into a pre-trained HOPE-Net deep neural network to obtain corresponding hand joint point position information; selecting a predetermined number of target RGB video frames from all the to-be-processed RGB video frames, and inputting the selected target RGB video frames into the I3D model for identification to obtain corresponding video frame features; inputting the hand joint point position information into an AGCN model to obtain corresponding position information features; and fusing the video frame features and the position information features in a one-to-one correspondence manner to obtain the probability of identifying the action instruction. The video frame is sequentially subjected to hand skeleton joint extraction and RGB and skeleton action feature extraction, and finally feature fusion is performed to obtain the action instruction probability, so that dependence on external hardware equipment is eliminated, and the method has strong robustness for illumination and scene change.

Description

technical field [0001] The present invention relates to the technical field of motion recognition, in particular to a method and device for motion recognition based on a first-person perspective. Background technique [0002] Although robots can understand human behavior intentions well and learn human behavior autonomously by learning actions from human demonstration videos, in practical applications, the learning of human behavior by robots requires a detailed understanding process, especially It is particularly challenging for robots to learn behaviors derived from daily activities. For example, in the first-person video taken by a wearable camera, the robot can only obtain the human hand's operation from a single angle. In this case, it is full of things such as fast hand movement and occlusion when the hand is operated, resulting in a large degree of unpredictability. Therefore, the recognition of subtle differences in human actions by robots, and the process of learni...

AI Technical Summary

Problems solved by technology

[0003] Based on the method of first-person perspective action recognition, the current methods mainly include three types: (1) use sensors such as LeapMtion to collect information about hand joints in demonstration videos, and then assist action recognition. This method requires hardware support and requires operation. Personnel need to demonstrate actions in a specific environment; (2) For the demonstration video, use dense trajectories to represent motion features, and use HOG to collect gesture features. This method is often disturbed by background and camera movement, and the calculation is relatively heavy (3) Segment the hands of the operator in the demonstration video and input them into the deep neural network for recognition. Although this method can effectively reduce background interference, most of the original information is missing

Obviously, the existing action recognition methods based on the first-person perspective have certain defects.

Images