Multi-modal fusion gesture recognition method based on color and depth information

Abstract

The present invention relates to a multi-modal fusion gesture recognition method based on the color and depth information. The method comprises the steps of 1) simultaneously shooting a gesture by a color camera and a depth camera to obtain the original data of the color and the depth of the gesture; 2) based on the gaussian smoothing method, subjecting the original data of the color and the depth of the gesture to filtering to obtain the smooth data of the color and the depth of the gesture; 3) adopting the skin color and the depth as the priori knowledge, respectively segmenting the smooth data of the color and the depth to obtain a skin color segmentation region and a depth segmentation region; 4) fusing the pixel layers of the skin color segmentation region and the depth segmentation region based on dynamic weights so as to form a fused segmentation map; 5) extracting the features of the fused segmentation map by using a feature descriptor and forming a feature map; 6 ) encoding the feature map to form gesture codes; 7) recognizing the gesture codes to realize the gesture recognition. Compared with the prior art, the above method is higher in gesture recognition accuracy.

Description

technical field [0001] The present invention relates to human-computer interaction technology, in particular to gesture interaction technology, and in particular to a multimodal fusion gesture recognition method based on color and depth information. Background technique [0002] Meaning of Gesture Recognition: The process of identifying the meaning of gestures from data obtained by sensors. [0003] Difficulties in gesture recognition problems: 1) Gesture recognition in complex scenes, including complex background and complex foreground; 2) Gesture recognition under different lighting conditions, including strong light and low light; 3) Gesture recognition under occlusion and rotation, Including, obstacle occlusion, human body occlusion, and rotation of different postures of the same gesture. [0004] The main processing methods of gesture recognition: 1) in-depth representation and recognition of single modal information; 2) effective fusion and recognition of multiple mod...

AI Technical Summary

Problems solved by technology

[0005] Insufficiency of the single mode method: the commonly used single mode can usually only express gestures from one aspect, for example: the acceleration sensor can reflect the acceleration changes in different directions during the gesture change process, when the gesture appears static or similar motion trajectory , the acceleration data is similar and difficult to distinguish; the color camera can reflect the image distribution of the gesture, and it is difficult to distinguish when the gesture is affected by different ambient light or an object with similar skin color appears

Images