Video human body behavior recognition method and system based on multi-mode double-flow 3D network

Abstract

The invention discloses a video human body behavior recognition method and system based on a multi-mode double-flow 3D network. The method comprises the steps of generating a depth dynamic image sequence DDIS based on a depth video; generating a posture evaluation graph sequence PEMS based on the RGB video; respectively inputting the depth dynamic graph sequence and the posture evaluation graph sequence into a 3D convolutional neural network, constructing a DDIS flow and a PEMS flow, and obtaining respective classification results; and fusing the obtained classification results to obtain a final behavior recognition result. The method has the beneficial effects that the DDIS can well describe the human body motion and the contour of the interactive object in the long-term behavior video by modeling the local space-time structure information of the video. The PEMS can clearly capture the change of the human body posture and eliminate the interference of the messy background. The multi-mode double-flow 3D network architecture can effectively model the global space-time dynamic of the behavior video in different data modes, and has excellent recognition performance.

Description

technical field [0001] The invention relates to the technical field of human behavior recognition, in particular to a video human behavior recognition method and system based on a multimodal dual-stream 3D network. Background technique [0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art. [0003] Video-based human action recognition has attracted more and more attention in the field of computer vision in recent years due to its wide range of applications, such as intelligent surveillance, video retrieval, and elderly care. Compared with image classification, video action recognition is a more challenging task because of the high-dimensional nature of videos, and the temporal structure between consecutive frames can also provide important additional information. Therefore, spatio-temporal feature learning is of great significance for video-based action recognition. Spatial...

AI Technical Summary

Problems solved by technology

Therefore, CNN-based methods tend to predict behaviors based on scenes and objects, which makes them more susceptible to cluttered backgrounds.

Second, although 3D CNN has the significant advantage of simultaneously learning spatio-temporal features, it is usually only applied to RGB videos

RGB data is highly sensitive to color and lighting changes, occlusion and background clutter, and moreover, it is difficult to obtain more high-level cues, such as human pose and body contour information, only from RGB video

Images