A gesture recognition method based on Fourier descriptor

Abstract

A gesture recognition method based on Fourier descriptor is a method for recognizing gestures through Fourier descriptors and artificial neural networks, including the steps of: reading a human hand image data set; detecting skin color and binary images; tracking and searching for the human hand contour; recalculating the horizontal and vertical coordinates of the human hand contour according to the coordinate conversion relationship to establish a new human hand contour; describing the new human hand contour with complex numbers, and applying Fourier transform to calculate the Fourier descriptor, and normalizing the Fourier descriptor; taking a number of normalized Fourier descriptors as gesture features, and training a set of weights and bias values ​​through an artificial neural network; using the artificial neural network containing the weights and bias values ​​to recognize the gesture to be recognized. The present invention can recognize gestures of the same shape but different rotation angles, and has the characteristics of superior performance and strong robustness.

Description

Technical Field

[0001] This research involves the fields of computer vision and artificial intelligence technology, and in particular, involves a gesture recognition method based on Fourier descriptors. Background Art

[0002] Since the invention of the car, the way people interact with the car has been constantly changing. Voice interaction is currently a common way of intelligent human-car interaction, but its user experience is not friendly enough for people with language barriers such as the deaf, children and the elderly. With the continuous development of gesture recognition technology, gesture recognition has also become a new way of human-car interaction in cars. Through gesture recognition technology, drivers and passengers can use different gestures to interact with on-board devices to enjoy a variety of functions, such as hanging up the phone, playing music, opening and closing windows, and adjusting air conditioning temperature, seat position and other functions. Since human hand postures are ever-changing and easily affected by factors such as light, skin color, background, etc., accurate, real-time and stable recognition of gestures has become a key problem to be solved.

[0003] A gesture recognition method based on Fourier descriptor mainly uses Fourier descriptor to describe the shape of gesture contours, and uses artificial neural network to classify the shape of gesture contours. Since Fourier descriptors have characteristics such as rotation invariance, gesture contours of the same shape but different rotation angles are classified as the same gesture. Such a method only focuses on the shape of gesture contours and omits the rotation direction of gesture contours. As far as vehicle-mounted equipment control is concerned, the rotation direction of gesture contours is of great significance to user experience. For example, the gesture of making a fist and extending the index finger, the index finger to the left and right can intuitively represent raising and lowering the air conditioning temperature or raising and lowering the volume. In response to such problems, this patent proposes a calculation method that can distinguish the rotation direction of gesture contours, so as to identify hand postures, with the advantages of high recognition rate, superior performance and good robustness. Summary of the invention

[0004] The object of the present invention is to provide a gesture recognition method based on Fourier descriptors to solve the problems of gesture classification, real-time performance and robustness that are often encountered in the gesture recognition process.

[0005] To achieve the above object, the present invention provides a gesture recognition method based on Fourier descriptor, which mainly includes the following steps:

[0006] ① Read out images of different gesture action datasets and extract gesture contours;

[0007] ② According to the coordinate conversion relationship, recalculate the coordinates of the gesture contour points and establish a new gesture contour;

[0008] ③ Calculate the Fourier descriptor of the new gesture contour;

[0009] ④Using the Fourier descriptor as the gesture action feature, a gesture recognition neural network model is obtained through artificial neural network training, and the gesture action to be recognized is recognized based on the gesture recognition neural network model.

[0010] For the gesture recognition method based on Fourier descriptor, the gesture contour extraction step in step ① is as follows:

[0011] ⑴ Convert the RGB color space image into an HSV color space image;

[0012] ⑵ Extract the skin color using the distribution characteristics of the skin color in the HSV color space to obtain a skin color image;

[0013] ⑶ Binarize the skin color image to obtain a binary image;

[0014] ⑷ Search for the hand boundary in the binary image through the boundary tracking method, and the hand boundary is the gesture contour.

[0015] For the gesture recognition method based on Fourier descriptor, the new gesture contour establishment step in step ② is as follows:

[0016] ⑴ Let (contourX(i), contourY(i)) be the horizontal and vertical coordinate values of the i-th contour point of the gesture contour, 0 <= i < L, where L is the number of gesture contour points;

[0017] ⑵ The new and old coordinate conversion formula: newContourX(i) = contourX(i), newContourY(i) = contourX(i) + contourY(i);

[0018] ⑶ (newContourX(i), newContourY(i)) are the horizontal and vertical coordinate values of the i-th contour point of the new gesture contour;

[0019] ⑷ All (newContourX, newContourY) contour points form the new gesture contour.

[0020] For the gesture recognition method based on Fourier descriptor, the Fourier descriptor calculation step of the new gesture contour in step ③ is as follows:

[0021] ⑴

[0022] ⑵ In the formula, s(i) is the complex form of the i-th contour point of the new gesture contour;

[0023] ⑶ ;

[0024] ⑷ In the formula, a is called the Fourier descriptor, which is the value after transforming s using the Fourier transform;

[0025] ⑸ ;

[0026] ⑹ is the modulus of the w-th Fourier descriptor, F(w) is the w-th normalized Fourier descriptor, w value is 1 to N, N takes a uniform value, and the value range is [8,L). The normalized Fourier descriptor is the Fourier descriptor of the new gesture contour.

[0027] In the method for gesture recognition based on Fourier descriptors, the step ④ uses the 1st to Nth F(w) of the new gesture profile as BP artificial neural network input, uses the BP artificial neural network to train the different gesture action data set images, and obtains a set of trained artificial neural network weights and bias weights. The BP artificial neural network containing the artificial neural network weights and bias weights is a gesture recognition neural network model, and the gesture recognition neural network model is used to recognize gesture actions.

[0028] Through the above scheme, the beneficial effects of the present invention are: it can distinguish gestures with the same shape but different rotation angles, can recognize more gesture actions; and has the advantages of recognition performance and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the gesture training of the present invention.

[0030] Figure 2 This is a flow chart of gesture recognition of the present invention.

[0031] Figure 3 This is a schematic diagram of a predefined gesture of the present invention.

[0032] Figure 4 It is a schematic diagram of the gesture outline of the present invention with the same shape but different rotation angles. DETAILED DESCRIPTION

[0033] The following is a complete description of the technical solution of the present invention to those skilled in the art in conjunction with the accompanying drawings of the present invention, so that the technical content of the present invention is easier to understand, and its purpose is to enable developers to develop new products based on the present solution. Any changes made in accordance with the spirit of the present invention are within the scope of protection of the present invention.

[0034] Implementation example - gesture training.

[0035] The gesture training process is as follows Figure 1 shown.

[0036] Step 1: Read the hand image dataset. This implementation example is based on OpenCV. OpenCV is an open-source cross-platform computer vision library that can run on systems such as Windows, Linux, and Android, and supports development languages such as Java and Python. The hand image dataset predefines 23 gestures, with 1 image for each gesture. The images are PNG files with a size of 640*480 and in RGB color. The 23 gestures predefined in the hand image dataset are as shown in Figure 3 . Read the hand image dataset file by calling the imread method in OpencCV.

[0037] Step 2: Extract the skin color. Convert the color space of the hand image from the RGB space to the HSV space by calling the cvtColor method in OpenCV. Set the lower threshold [100, 20, 20] and the upper threshold [135, 255, 255]. Search for the skin color in the HSV hand image by calling the inRange method in OpenCV, and create a new binary hand image. Represent the skin color (hand) in white and the non-skin color (background) in black.

[0038] Step 3: Extract the gesture contour. Find the contour of the binary hand image by calling the findContours method in OpenCV. The contour is the gesture contour.

[0039] Step 4: Traverse the gesture contour, read the horizontal and vertical coordinates (x(i), y(i)) of the i-th contour point, convert the horizontal and vertical coordinates into new horizontal and vertical coordinates, newContourX(i) = x(i), newContourY(i) = x(i) + y(i), 0 <= i < L, where L is the number of contour points that make up the gesture contour. The contour composed of all new contour points is called the new gesture contour.

[0040] Step 5: Calculate the Fourier descriptor of the new gesture contour. First, describe the horizontal and vertical coordinates of the new contour points with the complex number s(i), , where 0 <= i < L. Then, perform a Fourier transform on s using the fft method in OpenCV. Next, shift the low-frequency data after the Fourier transform to the center position by calling the fftshift method in OpenCV.截取 N / 2 low-frequency and high-frequency data from both sides of the center position, where N is 16. Finally, shift the low-frequency and high-frequency data back to the original position by calling the ifftshift method in OpenCV. The low-frequency and high-frequency data shifted back to the original position is the Fourier descriptor a(k), 0 <= k <= N.

[0041] Step 6: Normalize the Fourier descriptor of the new gesture contour. Then, the modulus of the Fourier descriptor is calculated based on the complex modulus formula. , normalize the model, , the F(w) is the normalized Fourier descriptor of the new gesture contour.

[0042] Step 7: Use BP artificial neural network to train the hand image dataset. The BP artificial neural network is divided into an input layer, a hidden layer and an output layer. The input layer is composed of N neurons, the hidden layer is composed of 2*N neuron groups, and the output layer is composed of 23 neurons. The input layer neurons accept the F(w), and the output layer neurons correspond to the predefined 23 gestures. The initial values ​​of weights and biases are random numbers, the activation function uses the sigmoid function, and the output layer uses the softmax function for decision. After 100,000 iterations of training, the BP artificial neural network adapts to the hand image dataset and obtains a set of weights and bias values ​​that can recognize the predefined 23 gestures. The BP artificial neural network containing the weights and bias values ​​is the gesture recognition neural network model.

[0043] Step 8 converts the weight and bias values ​​into JSON data, and stores the JSON data in a text file.

[0044] Implementation Example - Gesture Recognition.

[0045] The gesture recognition process is as follows Figure 2 shown.

[0046] Step 1 reads the weight and bias values ​​from the text file, and uses the weight and bias values ​​to initialize the weight and bias values ​​of the gesture recognition neural network model.

[0047] Step 2: Capture the camera image. Read the camera image by calling the VideoCapture and read methods in OpencCV.

[0048] Step3 Extract skin color. Convert the color space of the hand image from RGB space to HSV space by calling the OpenCV method cvtColor. Set the lower threshold [100, 20, 20] and the upper threshold [135, 255, 255], search for the skin color in the HSV hand image by calling the OpenCV method inRange, and create a new binary hand image, using white to represent skin color (hand) and black to represent non-skin color (background).

[0049] Step 4: Extract the gesture contour. Find the gesture contour of the binary hand image by calling the method findContours in OpenCV.

[0050] Step 5: Traverse the gesture contour, read the horizontal and vertical coordinates (x(i), y(i)) of the i-th contour point, convert the horizontal and vertical coordinates into new horizontal and vertical coordinates, newContourX(i) = x(i), newContourY(i) = x(i) + y(i), 0 <= i < L, where L is the number of contour points that make up the gesture contour, and (newContourX(i), newContourY(i)) are the horizontal and vertical coordinates of the new contour points. The contour composed of all new contour points is called the new gesture contour.

[0051] Step 6: Calculate the Fourier descriptor of the new gesture contour. First, describe the horizontal and vertical coordinates of the new contour points with the complex number s(i), , where 0 <= i < L; then use the method fft in OpenCV to perform the Fourier transform on s; then use the method fftshift in OpenCV to move the low-frequency data after the Fourier transform to the center position, intercept N / 2 low-high frequency data from both sides of the center position, where N is taken as 16; finally, use the method ifftshift in OpenCV to move the low-high frequency data back to the original position, and the low-high frequency data moved back to the original position is the Fourier descriptor a(k), 0 <= k <= N.

[0052] Step 7: Normalize the Fourier descriptor of the new gesture contour. Immediately calculate the modulus of the Fourier descriptor based on the complex modulus formula , and perform normalization processing on the modulus, , where F(w) is the normalized Fourier descriptor of the new gesture contour.

[0053] Step 8: Recognize the gesture. Use F(w) as the input of the gesture recognition neural network model, and recognize the gesture action through the gesture recognition neural network model.

[0054] Judge whether to end. If not, jump to Step 2; otherwise, end.

Claims

1. A gesture recognition method based on Fourier descriptors mainly includes the following steps: ① Read images of different gesture action datasets and extract gesture contours; ② According to the coordinate transformation relationship, recalculate the coordinates of gesture contour points to establish a new gesture contour. The coordinate transformation refers to the transformation formula newContourX(i) = contourX(i) and the transformation formula newContourY(i) = contourX(i) + contourY(i), where (contourX(i), contourY(i)) are the horizontal and vertical coordinate values of the i-th contour point of the gesture contour, 0 <= i < L, where L is the number of gesture contour points, and (newContourX(i), newContourY(i)) are the horizontal and vertical coordinate values of the i-th contour point of the new gesture contour. All (newContourX, newContourY) contour points form the new gesture contour; ③ Calculate the Fourier descriptors of the new gesture contour; ④ Use the Fourier descriptors as gesture action features, train through an artificial neural network to obtain a gesture recognition neural network model, and recognize the gesture action to be recognized based on the gesture recognition neural network model.

2. The gesture recognition method based on Fourier descriptor according to claim 1, characterized in that: The gesture contour extraction step in step ① is as follows: ⑴ Convert the RGB color space image into an HSV color space image; ⑵ Extract skin color using the distribution characteristics of skin color in the HSV color space to obtain a skin color image; ⑶ Binarize the skin color image to obtain a binary image; ⑷ Search for the hand boundary in the binary image through the boundary tracking method, and the hand boundary is the gesture contour.

3. The gesture recognition method based on Fourier descriptor according to claim 1, characterized in that: The calculation step of the Fourier descriptors of the new gesture contour in step ③ is as follows: ⑴ s(i) = newContourX(i) + j·newContourY(i), i = 0, 1,......, L - 1 ⑵ In the formula, s(i) is the complex form of the i-th contour point of the new gesture contour; ⑶ ⑷ In the formula, a is called the Fourier descriptor, which is the value after transforming s using the Fourier transform; ⑸ Where w=1,2,3,......,N and 8<=N <L ⑹ ||a(w)|| is the modulus of the w-th Fourier descriptor, F(w) is the w-th normalized Fourier descriptor, the value of w is from 1 to N, N takes a unified value, and the value range is [8, L). The normalized Fourier descriptor is the Fourier descriptor of the new gesture contour.

4. The gesture recognition method based on Fourier descriptor according to claim 3, characterized in that: In step ④, the first to N-th F(w) of the new gesture contour are used as the input of the BP artificial neural network, and the BP artificial neural network is used to train the images of different gesture action datasets to obtain a set of trained artificial neural network weights and bias values. The BP artificial neural network containing the artificial neural network weights and bias values is the gesture recognition neural network model, and the gesture recognition neural network model is used to recognize gesture actions.

Images