Android human-computer interaction method based on MediaPipe gesture recognition model

Abstract

An Android human-computer interaction method based on the MediaPipe gesture recognition model includes: S1: turn on the mobile phone camera to obtain each frame of the hand image; S2: input the hand image obtained in S1 into the gesture recognition model, and return the position information of the key points of the hand in real time; S3: determine what action to make based on the key point information obtained in S2, and obtain action information; S4: convert the action information obtained in S3 into an operation instruction of the mobile phone, and operate the mobile phone in real time. The present invention adopts gesture recognition technology based on computer vision, uses the front camera of the mobile phone to collect gesture information, realizes human-computer interaction, and realizes an Android human-computer interaction method based on the MediaPipe gesture recognition model. The Android human-computer interaction method based on the MediaPipe gesture recognition model provided by the present invention enables users to operate mobile phones in the air without touching the screen, and is suitable for application scenarios where it is inconvenient for users to touch the mobile phone screen with both hands.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and relates to a gesture recognition algorithm, and in particular to an Android human-computer interaction method based on a MediaPipe gesture recognition model. Background Art

[0002] In recent years, with the development of society, human-computer interaction has gradually become an indispensable part of daily life. In daily life, communication between people is usually supplemented by gestures to convey some information or express a specific intention. As a means of interaction that conforms to human daily habits, it is the most intuitive and natural way of interpersonal communication. Gesture recognition is an indispensable key technology for the new generation of human-computer interaction. At present, there are two directions of gesture recognition: the first is a gesture recognition system with data gloves as input devices. This direction requires the interactor to wear data gloves or sensor-like terminal devices to obtain data information, such as joysticks in VR games, handles in game consoles, etc.; the second is a gesture recognition system with the help of computer vision, which uses cameras as input devices. The input device can use single or multiple cameras or cameras to collect gesture information, and the images obtained by computer system analysis are used to recognize gestures, such as Baidu gesture recognition technology, ArcSoft gesture recognition technology, Google gesture recognition technology, etc.

[0003] Gesture recognition technology is simpler than other input devices. It only requires users to complete simple actions through gestures to express rich meanings. It is also easy to install and generally only requires a single camera as an input device.

[0004] Due to the deep number of neural network model layers and huge number of parameters, the existing gesture recognition technology currently deployed on mobile phones is basically unable to achieve real-time detection. At the same time, some gesture recognition technologies also require additional storage space. By interacting with the additional storage space, the usage delay is further increased, making the user experience extremely poor. Summary of the invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide an Android human-computer interaction method based on the MediaPipe gesture recognition model.

[0006] The present invention adopts gesture recognition technology based on computer vision, utilizes the front camera of a mobile phone to collect gesture information, realizes human-computer interaction, and realizes an Android human-computer interaction method based on the MediaPipe gesture recognition model.

[0007] The present invention obtains an image of the hand by turning on the mobile phone camera, and processes the image in real time through a gesture recognition model to obtain corresponding gesture information, and processes the recognized real-time movement information of the hand into various command information, thereby achieving the purpose of operating the mobile phone without touching the mobile phone screen.

[0008] The present invention provides an Android human-computer interaction method based on the MediaPipe gesture recognition model, comprising the following steps:

[0009] S1: Turn on the mobile phone camera and obtain each frame of the hand image;

[0010] S2: Input the hand image obtained in step S1 into the gesture recognition model, and return the position information of the key points of the hand in real time;

[0011] S3: for the key point information obtained in step S2, determine what action to take and obtain action information;

[0012] S4: The action information obtained in step S3 is converted into an operation instruction of the mobile phone, and the mobile phone is operated in real time.

[0013] Furthermore, the gesture recognition model described in step S2 is divided into two parts, a palm detection model and a hand key point detection model, specifically including:

[0014] S2-1: Input a single frame image into the palm detection model to detect whether there is a hand in the image. If so, obtain the square bounding box of the palm. The palm detection model is an Encoder-Decoder architecture. The Encoder is a 6-layer convolutional neural network that encodes the image information into a low-dimensional vector representation. The Decoder is a 3-layer neural network that decodes the low-dimensional vector representation and finally represents 11136 anchor point information.

[0015] S2-2: After palm detection of the entire image, the square bounding box of the palm position is used as a fixed position, and the bounding box of the entire hand is cut out, which is input into the hand key point detection model to obtain the precise key point positioning of 21 2D finger joint coordinates in the hand area. The hand key point detection model is a regression model. First, the real image and the generated image are convolved into a 256*256 RGB image, which is then input into the feature extraction neural network to extract the precise key point positioning of 21 2D finger joint coordinates in the hand area;

[0016] Furthermore, the process of determining what action to take and obtaining action information in step S3 specifically includes:

[0017] S3-1: Set the flag to detect whether an action has been taken. If flag = False, it is determined that no action has been taken, and the process goes to S3-2. If flag = True, it is determined that an action has been taken, and the process goes to S3-5.

[0018] S3-2: Set flag to flag = True, which means that an action is performed. Loop to obtain the location of the knuckle key points of each frame, calculate the offset of the key points between the two frames, loop 15 times, continuously add the offsets, and obtain the final offset offset. If the offset is greater than the specified offset threshold, it means that a dynamic action is performed, and go to S3-3. If the offset is less than the specified offset threshold, it means that a static action is performed, and go to S3-4.

[0019] The formula for the superposition offset is as follows:

[0020]

[0021] Among them, offset is the final offset, P i Represents key point positioning;

[0022] S3-3: If offset>0, it means that the palm slides to the right and performs a right sliding action. If offset<0, it means that the palm slides to the left and performs a left sliding action.

[0023] S3-4: Loop to obtain the knuckle key point positioning of each frame, input it into the static gesture recognition model, obtain the static gesture, loop 15 times, if the last 5 static gestures are the same, then take this static gesture as the final action;

[0024] S3-5: When the palm detection model detects that there is no hand in the image, set the parameters to flag=False, offset=0;

[0025] Furthermore, the static gesture recognition model described in step S3-4 specifically includes:

[0026] S3-4-1: Obtain accurate key point locations of 21 2D finger joint coordinates in the hand area through the hand key point detection model, store the key point locations in a file, and annotate them with action labels;

[0027] S3-4-2: Take the key point locations from the file, transform and standardize the coordinates, and process the data. The specific formula is as follows:

[0028]

[0029]

[0030] Among them, P represents 21 2D finger joint coordinates, P (i,1) represents the first coordinate of the i-th finger joint, P (i,2) Represents the second coordinate of the i-th finger joint;

[0031] S3-4-3: Input the processed data into the static gesture recognition model to obtain static gestures. The static gesture recognition model is a 4-layer multi-layer perceptron. The first layer is the input layer, the second layer and the third layer are two hidden layers. Each layer uses dropout to avoid overfitting, and uses ReLU as the activation function. The fourth layer is the output layer, and softmax is used as the activation function. Finally, the key point coordinate position information is extracted and converted into static gesture categories.

[0032] The present invention adopts gesture recognition technology based on computer vision, uses the front camera of a mobile phone to collect gesture information, realizes human-computer interaction, and realizes an Android human-computer interaction method based on the MediaPipe gesture recognition model. The Android human-computer interaction method based on the MediaPipe gesture recognition model provided by the present invention enables users to operate mobile phones in the air without touching the screen, and is suitable for application scenarios where users are inconvenient to touch the mobile phone screen with both hands.

[0033] The beneficial effects of the present invention are:

[0034] 1. The Android human-computer interaction method based on the MediaPipe gesture recognition model provided by the present invention realizes real-time detection, reduces usage delay, and improves user experience.

[0035] 2. The Android human-computer interaction method based on the MediaPipe gesture recognition model provided by the present invention does not need to store gesture features and occupy additional storage space, thereby improving operating efficiency and reducing latency.

[0036] 3. The Android human-computer interaction method based on the MediaPipe gesture recognition model provided by the present invention enables users to operate mobile phones in the air without touching the screen, which is suitable for application scenarios where it is inconvenient for users to touch the mobile phone screen with both hands. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the operation process of the gesture recognition model of the present invention;

[0038] Figure 2 is the palm detector model architecture of the present invention;

[0039] Figure 3 It is the hand key point detection model architecture of the present invention;

[0040] Figure 4 are the 21 key point positions of the hand of the present invention;

[0041] Figure 5 is an action acquisition flow chart of the present invention;

[0042] Figure 6 It is the static gesture recognition model architecture of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The present invention provides an Android human-computer interaction method based on the MediaPipe gesture recognition model, comprising the following steps:

[0045] S1: Turn on the mobile phone camera and obtain each frame of the hand image;

[0046] S2: Input the hand image obtained in step S1 into the gesture recognition model, and return the position information of the key points of the hand in real time; the gesture recognition model is divided into two parts, the palm detection model and the hand key point detection model, and its operation process is as follows Figure 1 As shown, the process is as follows:

[0047] S2-1: Input a single frame image into the palm detection model to detect whether there is a hand in the image. If so, obtain the square bounding box of the palm. The palm detection model is an Encoder-Decoder architecture. The Encoder is a 6-layer convolutional neural network that encodes the image information into a low-dimensional vector representation. The Decoder is a 3-layer neural network that decodes the low-dimensional vector representation and finally represents it as 11136 anchor point information. The palm detector model architecture is as follows: Figure 2 As shown;

[0048] S2-2: After palm detection of the entire image, the square bounding box of the palm is used as a fixed position to cut out the bounding box of the entire hand, which is input into the hand key point detection model to obtain the precise key point positioning of 21 2D finger joint coordinates in the hand area;

[0049] The hand key point detection model is a regression model. First, the real image and the generated image are convolved into a 256*256 RGB image, and then it is input into the feature extraction neural network to extract the precise key point positioning of 21 2D finger joint coordinates in the hand area. The architecture of the hand key point detection model is as follows: Figure 3 As shown;

[0050] Among them, 21 2D knuckle coordinates are accurate key points such as Figure 4 As shown, specifically including;

[0051]

[0052] S3: for the key point information obtained in step S2, determine what action to take and obtain action information; the process of determining what action to take and obtaining action information is as follows: Figure 5 As shown, the specific process is as follows:

[0053] S3-1: Set the flag to detect whether an action has been taken. If flag = False, it is determined that no action has been taken, and the process goes to S3-2. If flag = True, it is determined that an action has been taken, and the process goes to S3-5.

[0054] S3-2: Set flag to flag = True, which means that an action is performed. Loop to obtain the location of the knuckle key points of each frame, calculate the offset of the key points between the two frames, loop 15 times, continuously add the offsets, and obtain the final offset offset. If the offset is greater than the specified offset threshold, it means that a dynamic action is performed, and go to S3-3. If the offset is less than the specified offset threshold, it means that a static action is performed, and go to S3-4.

[0055] The formula for the superposition offset is as follows:

[0056]

[0057] Among them, offset is the final offset, P i Represents key point positioning;

[0058] S3-3: If offset>0, it means that the palm slides to the right and performs a right sliding action. If offset<0, it means that the palm slides to the left and performs a left sliding action.

[0059] S3-4: Loop to obtain the knuckle key point positioning of each frame, input it into the static gesture recognition model, obtain the static gesture, loop 15 times, if the last 5 static gestures are the same, then take this static gesture as the final action;

[0060] The static gesture recognition model specifically includes:

[0061] S3-4-1: Obtain accurate key point locations of 21 2D finger joint coordinates in the hand area through the hand key point detection model, store the key point locations in a file, and annotate them with action labels;

[0062] S3-4-2: Take the key point locations from the file, transform and standardize the coordinates, and process the data. The specific formula is as follows:

[0063]

[0064]

[0065] Among them, P represents 21 2D finger joint coordinates, P (i,1) represents the first coordinate of the i-th finger joint, P (i,2) Represents the second coordinate of the i-th finger joint;

[0066] S3-4-3: Input the processed data into the static gesture recognition model to obtain static gestures. The model is a 4-layer multi-layer perceptron. The first layer is the input layer, the second and third layers are two hidden layers, each layer uses dropout to avoid overfitting, and uses ReLU as the activation function. The fourth layer is the output layer, and softmax is used as the activation function. Finally, the key point coordinate position information is extracted and converted into static gesture categories. The model architecture is as follows: Figure 6 shown.

[0067] S3-5: When the palm detection model detects that there is no hand in the image, set the parameters to flag=False, offset=0;

[0068] S4: The action information obtained in step S3 is converted into an operation instruction of the mobile phone, and the mobile phone is operated in real time.

[0069] The present invention is based on computer vision gesture recognition technology, uses the front camera of a mobile phone to collect gesture information, realizes human-computer interaction, and realizes an Android human-computer interaction method based on the MediaPipe gesture recognition model. The Android human-computer interaction method based on the MediaPipe gesture recognition model provided by the present invention realizes real-time detection, reduces usage delay, and improves user experience; the Android human-computer interaction method based on the MediaPipe gesture recognition model provided by the present invention does not need to store gesture features and occupy additional storage space, thereby improving operating efficiency and reducing delay; the Android human-computer interaction method based on the MediaPipe gesture recognition model provided by the present invention enables users to operate the mobile phone gallery in the air without touching the screen, and is suitable for application scenarios where users are inconvenient to touch the mobile phone screen with both hands.

[0070] It should be understood that the specific embodiments described above are only used to explain the present invention, and are not used to limit the present invention. Obvious changes or modifications derived from the spirit of the present invention are still within the protection scope of the present invention.

Claims

1. The Android human-computer interaction method based on the MediaPipe gesture recognition model includes the following steps: S1: Turn on the mobile phone camera and obtain each frame of the hand image; S2: Input the hand image obtained in step S1 into the gesture recognition model, and return the position information of the key points of the hand in real time; the gesture recognition model is divided into two parts, a palm detection model and a hand key point detection model, which specifically includes: S2-1: Input a single frame image into the palm detection model to detect whether there is a hand in the image. If so, obtain the square bounding box of the palm. The palm detection model is an Encoder-Decoder architecture. The Encoder is a 6-layer convolutional neural network that encodes the image information into a low-dimensional vector representation. The Decoder is a 3-layer neural network that decodes the low-dimensional vector representation and finally represents 11136 anchor point information. S2-2: After palm detection of the entire image, the square bounding box of the palm position is used as a fixed position, and the bounding box of the entire hand is cut out, which is input into the hand key point detection model to obtain the precise key point positioning of 21 2D finger joint coordinates in the hand area. The hand key point detection model is a regression model. First, the real image and the generated image are convolved into a 256*256 RGB image, which is then input into the feature extraction neural network to extract the precise key point positioning of 21 2D finger joint coordinates in the hand area; S3: for the key point information obtained in step S2, determine what action to take and obtain action information; the determining what action to take and obtaining action information specifically includes: S3-1: Set a flag to detect whether an action has been taken. If flag = False, it is determined that no action has been taken, and the process goes to step S3-2. If flag = True, it is determined that an action has been taken, and the process goes to step S3-5. S3-2: Set flag to flag = True, indicating that an action is performed, loop to obtain the location of the knuckle key points of each frame, calculate the offset of the key points between the two frames, loop 15 times, continuously add the offsets, and obtain the final offset offset. If the offset is greater than the specified offset threshold, it means that a dynamic action is performed, and go to step S3-3. If the offset is less than the specified offset threshold, it means that a static action is performed, and go to step S3-4. The formula for the superposition offset is as follows: Among them, offset is the final offset, P i Represents key point positioning; S3-3: If offset>0, it means that the palm slides to the right and performs a right sliding action. If offset<0, it means that the palm slides to the left and performs a left sliding action. S3-4: Loop to obtain the location of the knuckle key points in each frame, input it into the static gesture recognition model, obtain the static gesture, loop 15 times, if the last 5 static gestures are the same, then take this static gesture as the final action; the static gesture recognition model specifically includes: S3-4-1: Obtain accurate key point locations of 21 2D finger joint coordinates in the hand area through the hand key point detection model, store the key point locations in a file, and annotate them with action labels; S3-4-2: Take the key point locations from the file, perform coordinate transformation and standardization, and process the data. The specific formula is as follows: Among them, P represents 21 2D finger joint coordinates, P (i,1) represents the first coordinate of the i-th finger joint, P (i,2) Represents the second coordinate of the i-th finger joint; S3-4-3: Input the processed data into the static gesture recognition model to obtain static gestures. The static gesture recognition model is a 4-layer multi-layer perceptron. The first layer is the input layer, and the second and third layers are two hidden layers. Each layer uses dropout to avoid overfitting, and uses ReLU as the activation function. The fourth layer is the output layer, and softmax is used as the activation function. Finally, the key point coordinate position information is extracted and converted into static gesture categories. S3-5: When the palm detection model detects that there is no hand in the image, set the parameters to flag=False, offset=0; S4: The action information obtained in step S3 is converted into an operation instruction of the mobile phone, and the mobile phone is operated in real time.

Images