A gesture recognition control method for intelligent devices based on image recognition

Abstract

The present invention discloses a gesture recognition and control method for smart devices based on image recognition, comprising: 1) capturing a user image with a camera and identifying the user's identity and approximate hand position based on OpenCV2; 2) gray-scaling and binarizing the image information of the rectangular portion of the hand, calculating the center distance of the hand figure to determine the palm position, and analyzing the coordinates of the key points of the hand using a CNN model; 3) comparing the coordinates of the key points of the hand obtained in step 2) with the user's preset operation gestures using the cosine similarity method by calculating their two-dimensional angular relationship and other motion information, thereby performing gesture analysis; and 4) sending control instructions to the device based on the analyzed gesture according to the control logic set by the user, thereby completing the remote operation of the device through gestures. The present invention uses a neural network to identify hand position and posture, introduces inductive analysis of gestures, and uses the recognition results to quickly control smart devices.

Description

Technical Field

[0001] The present invention relates to the field of device technology, and in particular to a gesture recognition control method for an intelligent device based on image recognition. Background Art

[0002] Current smart device input methods primarily rely on keyboards, mice, and touchscreens. Traditional input devices may not be suitable in certain situations, such as when facing distance or when hands are wet or otherwise dirty. Existing gesture recognition systems primarily rely on sensors to identify the user's hand motion trajectory, perform relevant analysis, and output operation signals. This method can only identify simple trajectories and cannot analyze and provide feedback on the user's specific hand gestures.

[0003] Currently, existing methods for identifying specific gestures mostly classify gestures through pattern recognition, establish decision tree models by extracting the apparent features of gestures, and perform recognition and analysis on gestures; or use 3D modeling methods for simulation and comparison. If new gestures are added, the model needs to be retrained, making it difficult to modify the function.

[0004] As smart devices become increasingly prevalent, human-computer interaction is an essential part of everyday life. Traditional input devices may not be suitable in certain situations. For example, touchscreen interaction can be affected by distance limitations or by wet or dirty hands. Summary of the Invention

[0005] The purpose of the present invention is to provide a gesture recognition and control method for smart devices based on image recognition, which uses a neural network to identify hand position and posture, introduces an inductive analysis of gestures, and quickly controls the smart device through the recognition results.

[0006] The object of the present invention is achieved as follows: a gesture recognition control method for an intelligent device based on image recognition, comprising the following steps:

[0007] Step 1) Capture the user image through the camera and identify the user's identity and approximate hand position based on OpenCV2;

[0008] Step 2) grayscale and binarize the image information of the rectangular part of the hand, calculate the center distance of the hand image to determine the palm position, and use the CNN model to analyze the coordinates of the key points of the hand;

[0009] Step 3) The hand key point position coordinates obtained in step 2) are compared with the user's preset operation gestures using the cosine similarity method by calculating their two-dimensional angle relationship and other motion information to perform gesture analysis;

[0010] Step 4) Based on the gesture obtained by analysis, control instructions are sent to the device according to the control logic set by the user to complete the air operation of the device through gestures.

[0011] As a further limitation of the present invention, the step 1) specifically includes:

[0012] Step 1.1) The user captures a facial image through a camera, and the user's identity is determined by a face recognition model; if the user is detected to be an authorized user, step 1.2) is executed;

[0013] Step 1.2) Using the mediapipe model in OpenCV2, define a hand function to perform preliminary segmentation on the input image and obtain the position information of the user's hand rectangle. The obtained hand rectangle position information includes the coordinates of the upper left corner of the rectangle (landmarks(x, y)) and the size of the rectangle (shape(width, height).

[0014] As a further limitation of the present invention, the step 2) specifically includes:

[0015] Step 2.1) The original image of the hand rectangle in step 1.2) is cut and grayscaled using the visual psychology formula (1);

[0016] G(x,y)=r(x,y)*299+g(x,y)*587+b(x,y)*114 / 1000 (1)

[0017] Step 2.2) The pixel information of the width×height portion of the image is binarized by threshold segmentation to obtain a two-dimensional matrix M consisting of only 0 and 255 gray levels. w,h ;

[0018] The cutting threshold is calculated using the Renyi entropy calculation formula (2):

[0019]

[0020]

[0021] K=argmax(H A (k)+H B (k)) (2)

[0022] K is the binarization cutting threshold, and the binarization method is shown in formula (3);

[0023]

[0024] Step 2.3) For the binarized matrix M obtained in step 2.2) w,h, take the region with the largest area value of 0, and its 0th-order moment is expressed as m00 = ∫∫f(x,y)dxdy; calculate the central moment by formula (4) to predict the palm position coordinates [locate_x, locate_y];

[0025] m10=∑ x ∑ y xf(x,y)

[0026]

[0027]

[0028] Step 2.4) Use CNN to train a specific gesture recognition model. The grayscale information matrix obtained by processing the gesture image is used as the input of the network model. The convolution operation is performed through formula (5) to obtain the feature matrix diagram.

[0029]

[0030] Where x(n) is the excitation signal, h(n) is the unit sample response, and * indicates convolution. After the convolution layer, the RELU function is selected to introduce nonlinear features. Then, the average pooling method is used to reduce the number of training parameters. Finally, the result is obtained through the fully connected layer. Where p(y=j|x;θ) represents the probability of outputting category j of y, θ is the network parameter, and x is the feature vector;

[0031] Step 2.5) After training, the model parameters are saved and tested on the test data set. For all test data, when the test accuracy reaches above 95%, it can be kept for future use;

[0032] Step 2.6) Use the gesture recognition model parameters obtained in step 2.4) to load the network model in TensorFlow and obtain the coordinates of the key points of the hand in the input image at this time.

[0033] As a further limitation of the present invention, the step 3) specifically includes:

[0034] Step 3.1) Perform a two-dimensional analysis on the coordinates of the key hand points obtained in step 2.6), and use formula (6) to calculate the angle between the lines connecting each point through the vector relationship. The angle relationship is used to represent the two-dimensional characteristics of the hand posture, and the finger bending angle and the angle between adjacent fingers are calculated;

[0035]

[0036] Step 3.2) Predefine the two-dimensional features corresponding to each type of gesture, and analyze the static hand posture in the image by comparing the finger bending angle and the angle between adjacent fingers with the preset posture information;

[0037] Step 3.3) The hand gesture obtained in step 3.2) and the palm position coordinates obtained in step 2.3) are used as a set of data, and the recognition data within two seconds are periodically analyzed to obtain the hand gesture, palm position offset direction, offset, and movement speed;

[0038] Step 3.4) Estimate the distance between the user's hand and the camera based on the size of the hand area obtained by recognition, standardize the offset direction, offset, and movement speed of the palm position, and obtain the final judgment vector; use the cosine similarity calculation method, as shown in formula (8), to calculate the similarity between the judgment vector and the preset standard vectors of various operation gestures, and judge the specific dynamic gesture or combined gesture; where A and B are the judgment vector and the standard vector respectively, and the value of its component A is i , B i are the products of each parameter and weight respectively, where the static gesture takes 1 if the value is the same as that in the standard vector, otherwise it takes 0;

[0039]

[0040] As a further limitation of the present invention, step 4) specifically includes: outputting a control signal to complete the air operation of the device through gestures; performing corresponding operations according to the gestures obtained in step 3.4), and directly issuing relative control instructions to the smart device according to the interactive interface provided by different operating systems.

[0041] As a further limitation of the present invention, the step 3.2) of comparing the finger bending angle and the angle between adjacent fingers with the preset posture information is specifically as follows: a set of finger bending angles and finger spacing angles obtained in step 3.1) are combined into a vector A with a dimension of 9 n =(xn1,xn2,xn3,……,xn9), calculate their Euclidean distance. The Euclidean distance calculation method between vectors A1 and A2 is shown in formula (7):

[0042]

[0043] After the gesture features recognized by the camera are compared with all predefined gestures, the gesture with the smallest Euclidean distance and no greater than the confidence threshold is selected as the judgment result gesture. The confidence threshold is determined according to the recognition sensitivity set by the user.

[0044] The present invention adopts the above technical solution, and compared with the existing technology, it has the following beneficial effects: the present invention combines image recognition and device control. First, the image information collected by the camera is segmented using a neural network-based segmentation model to segment the hand area, tracking the hand position and image information to further identify gesture usage. On this basis, the specific coordinates of each key point of the hand are identified. The hand posture is identified and summarized by analyzing the two-dimensional features of the obtained hand key point information. The user's hand posture is compared with predefined posture information to determine the user's hand posture, and the corresponding device operation is performed. This method has high accuracy and customization space, and has a wide range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the process of the present invention.

[0046] Figure 2 Screenshot of gesture control feedback in the present invention.

[0047] Figure 3 Schematic diagram of the two-dimensional characteristic angles of the hand in the present invention. DETAILED DESCRIPTION

[0048] like Figure 1 The method for controlling gesture recognition of an intelligent device based on image recognition includes the following steps:

[0049] Step 1) Capture the user image through the camera and identify the user's identity and approximate hand position based on OpenCV2;

[0050] Step 1.1) The user captures a facial image through a camera, and the user's identity is determined by a face recognition model; if the user is detected to be an authorized user, step 1.2) is executed;

[0051] Step 1.2) Using the mediapipe model in OpenCV2, define a hand function to perform preliminary segmentation on the input image and obtain the rectangular position information of the user's hand;

[0052] The obtained hand rectangle position information is:

[0053] Coordinates of the upper left corner of the rectangle landmarks(x,y)

[0054] Rectangle size shape (width, height)

[0055] Step 2) grayscale and binarize the image information of the rectangular part of the hand, calculate the center distance of the hand image to determine the palm position, and use the CNN model to analyze the coordinates of the key points of the hand;

[0056] Step 2.1) The original image of the hand rectangle in step 1.2) is cut and grayscaled using the visual psychology formula (1);

[0057] G(x,y)=r(x,y)*299+g(x,y)*587+b(x,y)*114 / 1000 (1)

[0058] Step 2.2) The pixel information of the width×height portion of the image is binarized by threshold segmentation to obtain a two-dimensional matrix M consisting of only 0 and 255 gray levels. w,h ;

[0059] The cutting threshold is calculated using the Renyi entropy calculation formula (2):

[0060]

[0061]

[0062] K=argmax(H A (k)+H B (k)) (2)

[0063] K is the binarization cutting threshold, and the binarization method is shown in formula (3).

[0064]

[0065] Step 2.3) For the binarized matrix M obtained in step 2.2) w,h , take the region with the largest area value of 0, and its 0th-order moment is expressed as m00 = ∫∫f(x,y)dxdy; Since the central moment of the image is an image feature with translation and rotation invariance, the central moment is calculated by formula (4) to predict the palm position coordinates [locate_x, locate_y];

[0066] m10=∑ x ∑ y xf(x,y)

[0067]

[0068]

[0069] Step 2.4) Use CNN to train a specific gesture recognition model. The grayscale information matrix obtained by processing the gesture image is used as the input of the network model. The convolution operation is performed through formula (5) to obtain the feature matrix diagram where f xy Indicates the value of the (x, y) position, b is the offset, is the value of the convolution kernel of channel c at position (p, q). After the convolution layer, the RELU function is selected to introduce nonlinear features, and then the average pooling method is used to reduce the number of training parameters. Finally, the result is obtained through the fully connected layer. Where p(y=j|x;θ) represents the probability of outputting category j of y, θ is the network parameter, and x is the feature vector;

[0070]

[0071] Where x(n) is the excitation signal, h(n) is the unit sample response, and * indicates convolution.

[0072] Step 2.5) After training, the model parameters are saved and tested on the test data set. For all test data, when the test accuracy reaches above 95%, it can be kept for future use;

[0073] Step 2.6) Use the gesture recognition model parameters obtained in step 2.4) to load the network model in TensorFlow and obtain the coordinates of the key points of the hand in the input image at this time.

[0074] Step 3) The hand key point position coordinates obtained in step 2) are compared with the user's preset operation gestures using the cosine similarity method by calculating their two-dimensional angle relationship and other motion information to perform gesture analysis;

[0075] Step 3.1) Perform a two-dimensional analysis on the coordinates of the key hand points obtained in step 2.6), and use formula (6) to calculate the angle between the lines connecting each point through the vector relationship. The angle relationship is used to represent the two-dimensional characteristics of the hand posture, and the finger bending angle and the angle between adjacent fingers (i.e., the finger spacing angle) are calculated.

[0076]

[0077] The finger bending angle finger_angle has 5 values, which correspond to the bending angles of the five fingers from the thumb to the little finger at the second knuckle near the fingertips; the finger spacing angle interfinger_angle has 4 values, which correspond to the angles formed by two adjacent fingers with the forked part as the vertex, such as Figure 3 shown.

[0078] Step 3.2) Predefine the two-dimensional features corresponding to each type of gesture, and analyze the static hand posture in the image by comparing the finger bending angle and the angle between adjacent fingers with the preset posture information;

[0079] For example Figure 3 The gesture "compare 2" in the figure is predefined as follows:

[0080] finger_angle≈[100°,180°,180°,30°,30°]

[0081] interfinger_angle≈[-45°,30°,0°,0°]

[0082] At this time, the finger_angle and interfinger_angle obtained by the camera in step 3.1) are compared with the defined gesture. The comparison method is: merge a set of finger_angle and interfinger_angle into a vector A with a dimension of 9 n =(xn1,xn2,xn3,……,xn9), calculate their Euclidean distance. The Euclidean distance between vectors A1 and A2 is calculated as shown in formula (7)

[0083]

[0084] After comparing the gesture features recognized by the camera with all predefined gestures, the gesture with the smallest Euclidean distance that is not greater than the confidence threshold is selected as the judgment result gesture; the confidence threshold is determined based on the recognition sensitivity set by the user.

[0085] Step 3.3) Use [gesture, [locate_x, locate_y]] (where gesture is the gesture obtained in step 3.2) and [locate_x, locate_y] is the palm position coordinates obtained in step 2.3) as a set of data. Periodically analyze the recognition data within 2 seconds to obtain the hand posture, palm position offset direction, offset, and movement speed;

[0086] If the gesture data collected by user A at time t1 is [NULL, [x1, y1]], and the next gesture data collected at time t2 is [gesture_id_7, [x2, y2]], then it can be known that user A made a gesture with ID 7 at this time, and the projection of the hand displacement on the two-dimensional plane is (x2-x1, y2-y1), and the offset is Moving speed v = d / (t2 - t1).

[0087] Step 3.4) Estimate the distance between the user's hand and the camera based on the size of the hand area identified, standardize the offset direction, offset, and movement speed of the palm position, and obtain the final judgment vector. Use the cosine similarity calculation method, see formula (8), to calculate the similarity between the judgment vector and the preset standard vectors of various operation gestures, and judge the specific dynamic gesture or combined gesture. Where A and B are the judgment vector and the standard vector respectively, and the value of its component A is i , B i are the products of each parameter and weight respectively, where the static gesture takes 1 if the value is the same as that in the standard vector, otherwise it takes 0.

[0088]

[0089] Step 4) According to the gesture obtained by analysis, the control logic set by the user is used to send a control instruction to the device to complete the operation of the device through the gesture. Figure 2 As shown;

[0090] Perform corresponding operations according to the gestures obtained in step 3.4), and directly issue corresponding control instructions to the smart device according to the interactive interfaces provided by different operating systems;

[0091] For example, if user A has pre-configured a "2" gesture and a left-hand gesture on their computer to shut down the computer, and an "OK" gesture to confirm the shutdown, the program will recognize the "2" gesture and wait for the next action. If it recognizes the "OK" gesture within the waiting time, the computer will execute the shutdown action; otherwise, the program will exit the recognition process.

[0092] This invention utilizes image recognition technology, boasting high accuracy. It can identify individual finger movements and complex gestures. Furthermore, it can recognize video streams and dynamic gesture changes. This method can be implemented on a variety of platforms. By converting two-dimensional angle information before further analysis, it allows for arbitrary definition of specific gestures and corresponding operations without requiring model retraining. This method has broad application prospects and can be embedded in various smart devices.

[0093] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solutions disclosed in the present invention, those skilled in the art can make some substitutions and modifications to some of the technical features therein according to the disclosed technical content without creative labor, and these substitutions and modifications are all within the protection scope of the present invention.

Claims

1. A gesture recognition control method for smart devices based on image recognition, characterized in that: The following steps are involved: Step 1) Capture the user image through the camera and identify the user's identity and approximate hand position based on OpenCV2; Step 1.1) The user captures a facial image through a camera, and the user's identity is determined by a face recognition model; if the user is detected to be an authorized user, step 1.2) is executed; Step 1.2) Using the mediapipe model in OpenCV2, define a hand function to perform preliminary segmentation on the input image and obtain the position information of the user's hand rectangle. The obtained hand rectangle position information includes the coordinates of the upper left corner of the rectangle (landmarks(x, y)) and the rectangle size (shape(width, height)). Step 2) grayscale and binarize the image information of the rectangular part of the hand, calculate the center distance of the hand image to determine the palm position, and use the CNN model to analyze the coordinates of the key points of the hand; Step 2.1) The original image of the hand rectangle in step 1.2) is cut and grayscaled using the visual psychology formula (1); G(x,y)=r(x,y) * 299+g(x,y) * 587+b(x,y) * 114 / 1000 (1) Step 2.2) The pixel information of the width×height portion of the image is binarized by threshold segmentation to obtain a two-dimensional matrix M consisting of only 0 and 255 gray levels. w,h ; The cutting threshold is calculated using the Renyi entropy calculation formula (2): K=argmax(H A (k)+H B (k)) (2) K is the binarization cutting threshold, and the binarization method is shown in formula (3); Step 2.3) For the binarized matrix M obtained in step 2.2) w,h , take the region with the largest area value of 0, and its 0th-order moment is expressed as m00 = ∫∫f(x,y)dxdy; calculate the central moment by formula (4) to predict the palm position coordinates [locate_x, locate_y]; m10=∑ x ∑ y xf(x,y) Step 2.4) Use CNN to train a specific gesture recognition model. The grayscale information matrix obtained by processing the gesture image is used as the input of the network model. The convolution operation is performed through formula (5) to obtain the feature matrix diagram. Where x(n) is the excitation signal, h(n) is the unit sample response, and * indicates convolution. After the convolution layer, the RELU function is selected to introduce nonlinear features. Then, the average pooling method is used to reduce the number of training parameters. Finally, the result is obtained through the fully connected layer. Where p(y=j|x;θ) represents the probability of outputting category j of y, θ is the network parameter, and x is the feature vector; Step 2.5) After training, the model parameters are saved and tested on the test data set. For all test data, when the test accuracy reaches above 95%, it can be kept for future use; Step 2.6) Use the gesture recognition model parameters obtained in step 2.4) to load the network model in TensorFlow and obtain the coordinates of the key points of the hand in the input image at this time; Step 3) The hand key point position coordinates obtained in step 2) are compared with the user's preset operation gestures using the cosine similarity method by calculating their two-dimensional angle relationship and other motion information to perform gesture analysis; Step 4) Based on the gesture obtained by analysis, control instructions are sent to the device according to the control logic set by the user to complete the air operation of the device through gestures.

2. The method for controlling gesture recognition of an intelligent device based on image recognition according to claim 1, characterized in that: The step 3) specifically includes: Step 3.1) Perform a two-dimensional analysis on the coordinates of the key hand points obtained in step 2.6), and use formula (6) to calculate the angle between the lines connecting each point through the vector relationship. The angle relationship is used to represent the two-dimensional characteristics of the hand posture, and the finger bending angle and the angle between adjacent fingers are calculated; Step 3.2) Predefine the two-dimensional features corresponding to each type of gesture, and analyze the static hand posture in the image by comparing the finger bending angle and the angle between adjacent fingers with the preset posture information; Step 3.3) The hand gesture obtained in step 3.2) and the palm position coordinates obtained in step 2.3) are used as a set of data, and the recognition data within two seconds are periodically analyzed to obtain the hand gesture, palm position offset direction, offset, and movement speed; Step 3.4) Estimate the distance between the user's hand and the camera based on the size of the hand area obtained by recognition, standardize the offset direction, offset, and movement speed of the palm position, and obtain the final judgment vector; use the cosine similarity calculation method, as shown in formula (8), to calculate the similarity between the judgment vector and the preset standard vectors of various operation gestures, and judge the specific dynamic gesture or combined gesture; where A and B are the judgment vector and the standard vector respectively, and the value of its component A is i， B i are the products of each parameter and weight respectively, where the static gesture takes 1 if the value is the same as that in the standard vector, otherwise it takes 0; 3. The method for controlling gesture recognition of an intelligent device based on image recognition according to claim 2, characterized in that: The step 4) specifically includes: outputting a control signal to complete the air operation of the device through gestures; performing corresponding operations according to the gestures obtained in step 3.4), and directly issuing relative control instructions to the smart device according to the interactive interface provided by different operating systems.

4. The method for controlling gesture recognition of an intelligent device based on image recognition according to claim 2, characterized in that: The step 3.2) is described as comparing the finger bending angle and the angle between adjacent fingers with the preset posture information as follows: the set of finger bending angles and finger spacing angles obtained in step 3.1) are combined into a vector A with a dimension of 9. n =(xn1,xn2,xn3,……,xn9), calculate their Euclidean distance. The Euclidean distance calculation method between vectors A1 and A2 is shown in formula (7): After the gesture features recognized by the camera are compared with all predefined gestures, the gesture with the smallest Euclidean distance and no greater than the confidence threshold is selected as the judgment result gesture. The confidence threshold is determined according to the recognition sensitivity set by the user.

Images