A method and apparatus for human behavior recognition based on multi-mode channel feature fusion

Abstract

The invention relates to a method and a device for recognizing human behavior based on multi-mode channel feature fusion. The method comprises the following steps: 1) constructing an adaptive hierarchical structure for an input depth image sequence; 2) extracting DMM features at each level of the adaptive hierarchical structure; 3) serially connecting DMM features of each level to construct adaptive depth motion map features as depth image channel features; 4) selecting relatively stable skeleton joints as reference joints according to the input skeleton image sequence; 5) calculating the displacement difference between the other joint nodes and the reference joint nodes in each frame as the feature expression in each frame; 6) combining the whole skeleton image sequence to obtain a feature expression of an action sequence as a skeleton image channel feature; 7) through feature fusion and classifying the fused feature, the human body behavior recognition result is obtained. The invention can describe the human body time sequence motion information and the spatial structure information at the same time, and has good recognition effect and robustness.

Description

technical field [0001] The present invention relates to a human body behavior recognition method and device based on multimodal feature fusion. First, a brand new descriptor is proposed from the depth sequence modulus: adaptive depth motion map, which is used to describe the time series motion information of the human body. Then, based on the skeleton sequence modulus, a joint point displacement descriptor is proposed to describe the spatial structure information of the human body. Finally, two fusion strategies are adopted: the decision-level fusion method and the feature-level fusion method, which fuse two feature descriptors for human behavior recognition. Background technique [0002] Human behavior recognition is an important research direction in the field of computer vision, and has important theoretical research significance. The purpose of the research is to automatically analyze the ongoing activity of a sequence of consecutive images in an unknown video. Human b...

AI Technical Summary

Problems solved by technology

However, there are inherent defects in a single model channel. The depth image is difficult to describe the spatial structure of the human body, and the skeleton image only loses a lot of details about the node position information.

This makes it challenging to extract more descriptive and discriminative behavioral descriptors from a single data model

Images