A gesture and line-of-sight fusion method for human-robot interaction and related devices

Abstract

This invention discloses a robot human-computer interaction method and related apparatus that integrates gestures and gaze. The method detects and tracks all people in each frame of an image and separates them to obtain the bounding boxes of all people. The pixel regions of each person's bounding box are cropped and input into a human pose estimation model one by one, outputting the key points of each person's body. The face orientation is determined based on the facial key points. Based on the facial key points, the pixel regions of the face are determined and cropped, and the face orientation is used as prior information and input together with the face pixel regions into the gaze estimation model to obtain the yaw and pitch angles of the face. Based on the yaw and pitch angles of the face, gaze attention is calculated to obtain the gaze attention result. When the face orientation and gaze attention result simultaneously meet preset requirements, the hand key points from the current moment and several consecutive frames before it are input frame by frame into a gesture recognition model, outputting the gesture type. This invention can more accurately determine the user's gesture intent.

Images