A gesture recognition method based on geometric characteristics

Abstract

A gesture recognition method based on geometric characteristics is a method for recognizing gestures according to the number of fingers and the finger direction angle, comprising the steps of: collecting a hand image by using a camera; searching for the skin color and binarizing the hand; removing the palm from the hand based on mathematical morphological erosion; obtaining the palm contour based on a contour search method; calculating the palm contour centroid and bottom center based on a moment; calculating the hand detection curve based on a trigonometric function; detecting the number of fingers based on the hand detection curve; calculating the finger direction angle based on a vector method; and recognizing gestures by combining the number of fingers and the finger direction angle. The present application can accurately, timely and stably recognize gestures, can recognize gestures of left and right tilting, forward and backward tilting and slightly curved fingers, and can recognize up to sixteen kinds of gestures.

Description

Technical Field

[0001] This research was supported by the Natural Science Foundation of Hainan Province (No: 621RC607) and involves the fields of image processing and pattern recognition technology, especially a gesture recognition method based on geometric characteristics. Background Technology

[0002] Due to their intuitive and natural characteristics, gestures have become an important means of human-computer interaction. With the development of artificial intelligence, gesture recognition technology has received widespread attention. Gesture recognition technology has been applied in fields such as vehicle control, smart homes, driver assistance systems, and sign language recognition, and it is of great significance for drivers and communication for the deaf and mute. Vision-based gesture recognition requires first capturing gesture images through a camera, and then recognizing the gestures. This is a challenging task for computers. Because human hand postures are highly variable and easily affected by factors such as lighting, skin color, and background, accurate, real-time, and stable gesture recognition has become a key problem to be solved.

[0003] Geometric feature-based gesture recognition methods primarily utilize features such as the number of fingers, angles, and relative positions to classify and distinguish gestures. However, due to individual differences and other factors, traditional geometric feature-based gesture recognition methods suffer from limitations in recognizing a limited number of gestures and computational complexity. To address these issues, this patent proposes a method for calculating the number of fingers and finger direction angles to identify hand postures. This method offers advantages such as simple computation, low computational load, high recognition rate, ability to recognize a wide range of gestures, and good robustness. Summary of the Invention

[0004] The objective of this invention is to provide a gesture recognition method based on geometric characteristics, which solves the problems of human hand differences, number of gestures, large computational load and robustness that are often encountered in the gesture recognition process.

[0005] To achieve the above objectives, the present invention provides a gesture recognition method based on geometric characteristics, which mainly includes the following steps:

[0006] ①Step 1: Use a color camera to capture a color image of a human hand;

[0007] ②Step 2: Based on skin color, search for the human hand from the color human hand image, create a new binary human hand image, using white to represent the human hand and black to represent the background;

[0008] ③Step 3: Based on the mathematical morphology algorithm, erode the fingers and separate the palm from the binary human hand image to create a new binary palm image;

[0009] ④ Step 4: Based on the contour extraction method, search for the palm contour from the binary palm image;

[0010] ⑤ Step 5: Calculate the centroid of the palm contour based on the zero-order moment and first-order moment calculation methods, and traverse the palm contour to find the lowest endpoint of the palm contour;

[0011] Step 6: Calculate the arc curve for hand detection based on trigonometric functions;

[0012] Step 7: Detect the pixel color of the binary hand image based on the arc curve of the hand detection, and count the number of fingers based on the color;

[0013] Step 8: Based on the vector angle calculation method, calculate the angle between the finger vector and the horizontal vector, where the angle is the finger direction angle;

[0014] ⑨ Step 9: Recognize gestures based on the combination of the number of fingers and the finger direction angle.

[0015] The gesture recognition method based on geometric characteristics, in step Step 6, divides the radian [-π / 6, π+π / 6] into N radian parts Ri, and calculates the Euclidean distance d between the centroid (centerX, centerY) of the palm contour and the bottom endpoint (buttomX, buttomY) of the palm contour; taking the centroid of the palm contour as the center, dist = d*k (k = 1.55-1.7) as the radius of the circle, and Ri as the radian, calculates the hand detection arc curve based on trigonometric functions. The hand detection arc curve is composed of N hand detection points DetectPointi, DetectPointi = (Xi, Yi), and the calculation formula is as follows:

[0016] Ri=i*(π+π / 3) / N, where 1<=i<=N, N=90 (1)

[0017] Yi=centerY - ⌊(dist * sin(Ri) )⌋ (2)

[0018] Xi=centerX + ⌊( dist * cos(Ri) )⌋ (3).

[0019] The gesture recognition method based on geometric characteristics, step 7, finger count detection, is as follows:

[0020] (1) Let fingerers represent the number of fingers, and set the initial value to fingerers=0; let continuity represent the number of consecutive occurrences of white pixels, and set the initial value to continuity=0; set the initial value to i=0.

[0021] (2) If i>=N, jump to (5); otherwise, read the coordinates (Xi,Yi) of the DetectPointi, detect whether the binary hand image pixel (Xi,Yi) is black. If it is black, set the number of consecutive occurrences of the white pixel to continuity=0 and jump to (4); otherwise, the number of consecutive occurrences of the white pixel is incremented.

[0022] (3) If the number of consecutive occurrences of the white pixel is != 3, then proceed directly to (4); otherwise, a new finger is detected, and the finger coordinates (fingersPOS) are stored. finger fingersPOS finger = (Xi, Yi), the number of fingers is finger++, jump to (4);

[0023] (4) i++, skip to (2) to continue the test;

[0024] (5) Output "fingers", which is the final number of fingers detected.

[0025] The gesture recognition method based on geometric characteristics includes the following steps in Step 8: calculating the finger direction angle:

[0026] (1) Let levelV be a horizontal vector pointing to the left, levelV = (-centerX, 0), fingerersPOSi (Xi, Yi) be the coordinates of the finger, fingerers be the number of fingers finally detected, angle be the finger direction angle, angle initial value = 0, i initial value = 0;

[0027] (2) If i>=fingers, then skip to (5); otherwise, take the centroid of the palm outline (centerX, centerY) as the starting point and the finger vector fingerVi as the ending point according to the vector formula.

[0028] (3) Calculate the angle α between the finger vector fingerVi and the horizontal vector levelV according to the formula for calculating the angle between two vectors. Calculate the angle sum angle = angle + a, i++, and jump to (2).

[0029] (4) Calculate angle = angle / fingers;

[0030] (5) Output angle, where angle is the final detected finger direction angle.

[0031] In the aforementioned gesture recognition method based on geometric characteristics, Step 9 divides [0, π] into several segments, compares the final detected finger direction angle value with the segment value, and obtains the finger direction value: left, upper left, up, upper right, and right pointing; (the number of fingers and the finger direction value) are the final recognized gesture value.

[0032] The beneficial effects of the present invention through the above scheme are as follows: it can accurately identify the number and direction of fingers, and recognize more than 16 kinds of gestures; it has depth recognition, and can recognize gestures that tilt left and right, tilt forward and backward, and slightly bend the fingers; it can reduce complexity and has good robustness. Attached Figure Description

[0033] Figure 1 This is a flowchart of the gesture recognition method of the present invention.

[0034] Figure 2 This is a schematic diagram of the geometric characteristics of gesture recognition in this invention. Detailed Implementation

[0035] The technical solution of the present invention will now be fully described to those skilled in the art in conjunction with the accompanying drawings, making the technical content of the present invention easier to understand. The purpose is to enable developers to develop new products based on the solution of the present invention. All modifications made according to the spirit and essence of the present invention fall within the protection scope of the present invention.

[0036] Implementation example.

[0037] Step 1: Capture a color image of a human hand using a color camera. The Rapoo C270L is a USB 2.0 color camera with a resolution of 1920*1080, a maximum frame rate of 30 frames per second, and is compatible with Android and Windows systems, supporting plug-and-play functionality. This implementation example is based on OpenCV, an open-source, cross-platform computer vision library that provides interfaces for languages ​​such as Python and includes common computer vision algorithm functions. The computer can acquire the color camera image by calling the VideoCapture method in OpenCV and convert the color camera image into a color image of a human hand with a resolution of 640*480.

[0038] Step 2 converts the color space of the colored hand image from RGB to HSV space by calling the cvtColor method in OpenCV. Using lower thresholds [100, 50, 0] and upper thresholds [125, 255, 255], the skin tone in the HSV colored hand image is searched by calling the inRange method in OpenCV, and a new binary hand image is created, using white to represent skin tone (hand) and black to represent non-skin tone (background).

[0039] Step 3 involves using a mathematical morphology algorithm to cut off the fingers and separate the palm. Using a 5th-order square matrix with all elements equal to 1 as the structuring element, the binary hand image is dilated and eroded using the OpenCV methods dilate and erode. The dilation is performed twice, followed by erosion 25 times, and finally another 25 dilations to obtain the palm image with the fingers cut off.

[0040] Step 4: Obtain the palm contour using a contour extraction method. First, use the `findContours` method in OpenCV to find the contour of the palm image. Finally, use the `contourArea` method in OpenCV to calculate the contour area; the contour with the largest area is the palm contour.

[0041] Step 5 calculates the centroid (centerX, centerY) of the hand contour based on the zeroth and first moments. The moments M of the hand contour can be calculated using the `moments` method in OpenCV, and the centroid can be calculated using `centerX = int(M['m10'] / M['m00'])` and `centerY = int(M['m01'] / M['m00'])`. Step 5 also calculates the bottom endpoint (buttomX, buttomY) of the hand contour by traversing the contour, where buttomX = centerX and buttomY is the maximum value in the Y direction of the hand contour point.

[0042] Step 6: Calculate N hand detection points (DetectPoint) based on trigonometric functions. i (X) i ,Y i ), where i = 0 to N-1, N = 90. The N hand detection points (DetectPoint) can be quickly calculated using the software NumPy. i (X) i ,Y i NumPy is an extension library for the Python language that supports large-dimensional arrays and matrix operations, providing a large library of mathematical functions for array operations. The linspace method in NumPy divides the radian [-π / 6, π+π / 6] into N radians Ri, where R = NumPy.linspace(NumPy.pi + NumPy.pi / 6, 0 - NumPy.pi / 6, N). Trigonometric function calculations can be quickly performed using the math library, a built-in mathematical function library provided by Python that supports integer and floating-point operations. The formulas for calculating Xi and Yi are shown below:

[0043] Xi = centerX + math.floor(dist * NumPy.cos(Ri)) (4)

[0044] Y i = centerY - math.floor(dist * NumPy.sin(Ri)) (5).

[0045] Step 7 traverses the human hand detection points DetectPointi(X i , Y i ) in sequence, detects the color of the binary human hand image pixels (X i , Y i ), and counts the number of fingers and records the finger coordinates according to the color. The specific process is as follows:

[0046] (1) Use fingers to represent the number of fingers, and set the initial value fingers = 0; use continuity to represent the number of consecutive occurrences of white pixels in the binary human hand image, and set the initial value continuity = 0; fingersPOS represents the set of finger coordinates, and set the initial value of fingersPOS to be empty; i represents the traversal order, and set the initial value i = 0; N is the length of the human hand detection points, N = 90;

[0047] (2) Detect whether the binary human hand image pixels (XXi, YYi) are black. If they are black, set the number of consecutive occurrences of the white pixel point continuity = 0, and jump to (4). Otherwise, the number of consecutive occurrences of the white pixel point continuity++, and jump to (3);

[0048] (3) If the number of consecutive occurrences of the white pixel point continuity == 3, confirm a new finger, increment the number of fingers finger++, and add the coordinates of the new finger fingersPOS i (XXi, YYi) to the finger coordinate set fingersPOS, and jump to (4). Otherwise, directly jump to (4);

[0049] (4) i++, if i < N, jump to (2) to continue the next detection point. Otherwise, jump to (5);

[0050] (5) Output fingers, and fingers is the finally detected number of fingers.

[0051] Step 8 calculates the angle between the finger vector and the horizontal vector based on the vector angle calculation method. The angle is the finger direction angle. The steps for calculating the angle are as follows:

[0052] (1) Let (centerX, centerY) be the centroid of the palm contour, levelV = (-centerX, 0) be the horizontal vector, angle be the finger direction angle, set the initial value of angle = 0, fingerers be the number of fingers finally detected, fingerersPOSi be the coordinate point of the i-th finger, set the initial value of i = 0;

[0053] (2) If i>=fingers, then skip to (5); otherwise, calculate the finger vector fingerVi, where fingerVi = (fingersPOSi.Xi - centerX, fingerersPOSi.Yi - centerY).

[0054] (3) Calculate the angle α between the finger vector fingerVi and the horizontal vector levelV according to the formula for calculating the angle between two vectors. Calculate the angle sum angle = angle + a, i++, and jump to (2).

[0055] (4) Calculate angle = angle / fingers;

[0056] (5) Output angle, which is the final calculated finger direction angle.

[0057] Step 9 identifies gestures based on the number of fingers and the finger angle. If 0...

Claims

1. A gesture recognition method based on geometric characteristics, mainly including the following steps: ①Step 1: Use a color camera to capture a color image of a human hand; ②Step 2: Based on skin color, search for the human hand from the color human hand image, create a new binary human hand image, using white to represent the human hand and black to represent the background; ③Step 3: Based on the mathematical morphology algorithm, erode the fingers and separate the palm from the binary human hand image to create a new binary palm image; ④ Step 4: Based on the contour extraction method, search for the palm contour from the binary palm image; ⑤ Step 5: Calculate the centroid of the palm contour based on the zero-order moment and first-order moment calculation methods, and traverse the palm contour to find the lowest endpoint of the palm contour; Step 6: Calculate the arc curve for hand detection based on trigonometric functions; Step 7: Detect the pixel color of the binary hand image based on the arc curve of the hand detection, and count the number of fingers based on the color; Step 8: Based on the vector angle calculation method, calculate the angle between the finger vector and the horizontal vector, where the angle is the finger direction angle; ⑨ Step 9: Recognize gestures based on the combination of the number of fingers and the finger direction angle.

2. The gesture recognition method based on geometric characteristics according to claim 1, characterized in that, Step ⑥ divides the radian [-π / 6, π+π / 6] into N radian Ri, and calculates the Euclidean distance d between the centroid (centerX, centerY) of the palm contour and the bottom endpoint (buttomX, buttomY) of the palm contour. Using the centroid of the palm contour as the center, dist = d*k (k = 1.55-1.7) as the radius, and Ri as the radian, a hand detection arc curve is calculated based on trigonometric functions. The hand detection arc curve consists of N hand detection points DetectPointi, where DetectPointi = (Xi, Yi). The calculation formula is shown below: Ri=i*(π+π / 3) / N, where 1<=i<=N, N=90 (1) Yi=centerY - ⌊(dist * sin(Ri) )⌋ (2) Xi=centerX + ⌊( dist * cos(Ri) )⌋ (3).

3. The gesture recognition method based on geometric characteristics according to claim 2, characterized in that, The steps for step ⑦, finger count detection, are as follows: (1) Let fingerers represent the number of fingers, and set the initial value to fingerers=0; let continuity represent the number of consecutive occurrences of white pixels, and set the initial value to continuity=0; set the initial value to i=0. (2) If i>=N, jump to (5); otherwise, read the coordinates (Xi,Yi) of the DetectPointi, detect whether the binary hand image pixel (Xi,Yi) is black. If it is black, set the number of consecutive occurrences of the white pixel to continuity=0 and jump to (4); otherwise, the number of consecutive occurrences of the white pixel is incremented. (3) If the number of consecutive occurrences of the white pixel is != 3, then proceed directly to (4); otherwise, a new finger is detected, and the finger coordinates (fingersPOS) are stored. finger fingersPOS finger = (Xi, Yi), the number of fingers is finger++, jump to (4); (4) i++, skip to (2) to continue the test; (5) Output "fingers", which is the final number of fingers detected.

4. The gesture recognition method based on geometric characteristics according to claim 3, characterized in that, The steps for calculating the finger direction angle in step ⑧ are as follows: (1) Let levelV be a horizontal vector pointing to the left, levelV = (-centerX, 0), fingerersPOSi (Xi, Yi) be the coordinates of the finger, fingerers be the number of fingers finally detected, angle be the finger direction angle, angle initial value = 0, i initial value = 0; (2) If i>=fingers, then skip to (5); otherwise, take the centroid of the palm outline (centerX, centerY) as the starting point and the finger vector fingerVi as the ending point according to the vector formula. (3) Calculate the angle α between the finger vector fingerVi and the horizontal vector levelV according to the formula for calculating the angle between two vectors. Calculate the angle sum angle = angle + a, i++, and jump to (2). (4) Calculate angle = angle / fingers; (5) Output angle, where angle is the final detected finger direction angle.

5. The gesture recognition method based on geometric characteristics according to claim 4, characterized in that, Step 9 divides [0, π] into several segments, compares the final detected finger direction angle value with the segment value, and obtains the finger direction value: left, upper left, up, upper right, and right pointing; (the number of fingers and the finger direction value) is the final recognized gesture value.

Images