A gesture recognition system and method based on VR glasses

Abstract

The present invention proposes a gesture recognition system and method based on VR glasses, wherein the gesture recognition system comprises a data acquisition module, a data processing module, a fingerprint recognition module and a gesture recognition module for acquiring first video data. The acquired first video data is processed, segmented and extracted to obtain a first hand area, and then the corresponding relationship between the fingerprint and the gesture recognition model is used to determine the first gesture recognition model according to the first fingerprint image information, and then the first gesture recognition model is used to perform gesture recognition on the first hand area to obtain a first gesture recognition result, which not only recognizes gestures quickly, but also ensures a high standard recognition rate.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a gesture recognition system and method based on VR glasses. Background Art

[0002] Virtual reality (VR) interactive technology is an emerging comprehensive information technology. It uses modern high technology with computer technology as the core to generate a realistic virtual environment of a specific range that integrates vision, hearing, and touch. Users can interact and influence each other with objects in the virtual environment in their own way with the help of necessary equipment, thereby generating feelings and experiences equivalent to being in the real environment. It integrates digital image processing, multimedia technology, computer graphics, sensor technology and other information technologies. It constructs a three-dimensional digital model through computer graphics, giving users a three-dimensional virtual environment visually.

[0003] VR gesture interaction means that users can control the content displayed on the display interface of virtual reality glasses or other terminals of virtual reality glasses through gestures. VR gesture interaction requires the terminal device to recognize the user's gestures. However, in the existing gesture recognition methods, there are problems such as slow recognition speed and inaccurate recognition results. Summary of the invention

[0004] Based on the above problems, the present invention proposes a gesture recognition system and method based on VR glasses. The system processes the acquired first video data, segments and extracts the first hand area, and then uses the correspondence between the fingerprint and the gesture recognition model to determine the first gesture recognition model according to the first fingerprint image information. The first gesture recognition model is then used to perform gesture recognition on the first hand area to obtain a first gesture recognition result. The system not only recognizes gestures quickly, but also ensures a high recognition rate.

[0005] In view of this, one aspect of the present invention proposes a gesture recognition system based on VR glasses, comprising: a data acquisition module for acquiring first video data, a data processing module, a fingerprint recognition module and a gesture recognition module; wherein,

[0006] The data acquisition module is configured to: acquire background image data and establish a background pixel library containing background sample pixel values;

[0007] The data processing module is configured as follows:

[0008] extracting a first video image sequence from the first video data;

[0009] Processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image;

[0010] Determine a connected domain in the first segmented image and mark the connected domain;

[0011] Determining a first hand region from the connected domain according to a preset hand region size range value;

[0012] The fingerprint recognition module is configured to: extract first fingerprint image information from the first hand area;

[0013] The gesture recognition module is configured as follows:

[0014] determining a first gesture recognition model according to the first fingerprint image information;

[0015] Performing gesture recognition on the first hand region using the first gesture recognition model to obtain a first gesture recognition result.

[0016] Optionally, in the step of processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image, the data processing module is specifically configured as follows:

[0017] Extracting current pixel points from the first video image sequence in a preset order;

[0018] Compare the pixel value of the current pixel with all the background sample pixel values ​​in the background pixel library;

[0019] Determine a first number of background sample pixel values ​​whose difference with the pixel value of the current pixel point is within a first preset range;

[0020] If the first number is greater than a first preset threshold, determining that the current pixel point belongs to a background point;

[0021] Repeat the above steps until the first video image sequence is compared and a current background point set is obtained;

[0022] Determining a background image region in the first video image sequence according to the current background point set;

[0023] The background image area is extracted from the first video image sequence to segment the background and the human body, thereby obtaining the first segmented image.

[0024] Optionally, after the step of determining a connected domain in the first segmented image and marking the connected domain and before the step of determining a first hand region from the connected domain according to a preset hand region size range value, the data processing module is configured to:

[0025] Compressing the first segmented image into an image with a pixel value of 64*64 and performing grayscale processing;

[0026] Input the processed image into the trained convolutional neural network model to obtain the face image area;

[0027] The face image area is deleted from the connected domain.

[0028] Optionally, in the step of extracting the first fingerprint image information from the first hand area, the fingerprint recognition module is specifically configured to:

[0029] extracting a first finger image from the first hand region;

[0030] Detecting and extracting a first fingerprint image from the first finger image;

[0031] Performing grayscale and normalization processing on the first fingerprint image to obtain a second fingerprint image;

[0032] Binarizing the second fingerprint image to obtain a third fingerprint image;

[0033] Performing enhancement and refinement processing on the third fingerprint image to obtain a fourth fingerprint image;

[0034] Extracting feature points from the fourth fingerprint image;

[0035] Matching the feature points with the feature points of fingerprint samples in a pre-stored fingerprint database;

[0036] When the matching value reaches the preset matching value, the information of the corresponding fingerprint sample is determined to be the first fingerprint image information.

[0037] Optionally, a control module is further included; after the step of performing gesture recognition on the first hand area using the first gesture recognition model to obtain a first gesture recognition result, the control module is configured to:

[0038] generating a first control instruction according to the first recognition result;

[0039] Sending the first control instruction to the terminal to be controlled;

[0040] Control and display the execution status of the first control instruction executed by the terminal to be controlled.

[0041] Another aspect of the present invention provides a gesture recognition method based on VR glasses, comprising:

[0042] Acquire background image data and establish a background pixel library containing background sample pixel values;

[0043] Acquire first video data;

[0044] extracting a first video image sequence from the first video data;

[0045] Processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image;

[0046] Determine a connected domain in the first segmented image and mark the connected domain;

[0047] Determining a first hand region from the connected domain according to a preset hand region size range value;

[0048] Extracting first fingerprint image information from the first hand area;

[0049] determining a first gesture recognition model according to the first fingerprint image information;

[0050] Performing gesture recognition on the first hand region using the first gesture recognition model to obtain a first gesture recognition result.

[0051] Optionally, the step of processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image includes:

[0052] Extracting current pixel points from the first video image sequence in a preset order;

[0053] Compare the pixel value of the current pixel with all the background sample pixel values ​​in the background pixel library;

[0054] Determine a first number of background sample pixel values ​​whose difference with the pixel value of the current pixel point is within a first preset range;

[0055] If the first number is greater than a first preset threshold, determining that the current pixel point belongs to a background point;

[0056] Repeat the above steps until the first video image sequence is compared and a current background point set is obtained;

[0057] Determining a background image region in the first video image sequence according to the current background point set;

[0058] The background image area is extracted from the first video image sequence to segment the background and the human body, thereby obtaining the first segmented image.

[0059] Optionally, after the step of determining a connected domain in the first segmented image and marking the connected domain and before the step of determining a first hand region from the connected domain according to a preset hand region size range value, the step further includes:

[0060] Compressing the first segmented image into an image with a pixel value of 64*64 and performing grayscale processing;

[0061] Input the processed image into the trained convolutional neural network model to obtain the face image area;

[0062] The face image area is deleted from the connected domain.

[0063] Optionally, the step of extracting first fingerprint image information from the first hand area includes:

[0064] extracting a first finger image from the first hand region;

[0065] Detecting and extracting a first fingerprint image from the first finger image;

[0066] Performing grayscale and normalization processing on the first fingerprint image to obtain a second fingerprint image;

[0067] Binarizing the second fingerprint image to obtain a third fingerprint image;

[0068] Performing enhancement and refinement processing on the third fingerprint image to obtain a fourth fingerprint image;

[0069] Extracting feature points from the fourth fingerprint image;

[0070] Matching the feature points with the feature points of fingerprint samples in a pre-stored fingerprint database;

[0071] When the matching value reaches the preset matching value, the information of the corresponding fingerprint sample is determined to be the first fingerprint image information.

[0072] Optionally, after the step of performing gesture recognition on the first hand region using the first gesture recognition model to obtain a first gesture recognition result, the method further includes:

[0073] generating a first control instruction according to the first recognition result;

[0074] Sending the first control instruction to the terminal to be controlled;

[0075] The execution status of the first control instruction executed by the terminal to be controlled is displayed.

[0076] According to the technical solution of the present invention, the gesture recognition system includes a data acquisition module, a data processing module, a fingerprint recognition module and a gesture recognition module for acquiring first video data. The acquired first video data is processed, segmented and extracted to obtain a first hand area, and then the first gesture recognition model is determined according to the first fingerprint image information using the corresponding relationship between the fingerprint and the gesture recognition model, and then the first gesture recognition model is used to perform gesture recognition on the first hand area to obtain a first gesture recognition result, which not only recognizes gestures quickly, but also ensures a high standard recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 is a schematic block diagram of a gesture recognition system based on VR glasses provided by an embodiment of the present invention;

[0078] Figure 2 This is a flow chart of a gesture recognition method based on VR glasses provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0080] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0081] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0082] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0083] Refer to the following Figure 1 to Figure 2 To describe a gesture recognition system and method based on VR glasses provided according to some embodiments of the present invention.

[0084] like Figure 1 As shown, an embodiment of the present invention provides a gesture recognition system based on VR glasses, including: a data acquisition module for acquiring first video data, a data processing module, a fingerprint recognition module and a gesture recognition module; wherein,

[0085] The data acquisition module is configured to: acquire background image data and establish a background pixel library containing background sample pixel values;

[0086] The data processing module is configured as follows:

[0087] extracting a first video image sequence from the first video data;

[0088] Processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image;

[0089] Determine a connected domain in the first segmented image and mark the connected domain;

[0090] Determining a first hand region from the connected domain according to a preset hand region size range value;

[0091] The fingerprint recognition module is configured to: extract first fingerprint image information from the first hand area;

[0092] The gesture recognition module is configured as follows:

[0093] determining a first gesture recognition model according to the first fingerprint image information;

[0094] Performing gesture recognition on the first hand region using the first gesture recognition model to obtain a first gesture recognition result.

[0095] It can be understood that the VR glasses in the embodiment of the present invention may include a data acquisition module, such as a camera. There may be multiple cameras. By using multiple cameras, better image data with more parameter information can be obtained; the data acquisition module may also include a touch screen, etc.

[0096] The data processing module may be a processing chip, such as a CPU, a GPU, etc.

[0097] The fingerprint recognition module is used to extract fingerprint data from the acquired image data and perform fingerprint recognition.

[0098] The gesture recognition module uses the correspondence between fingerprints and gesture recognition models to determine a first gesture recognition model according to the first fingerprint image information, and then uses the first gesture recognition model to perform gesture recognition on the first hand area to obtain a first gesture recognition result.

[0099] The technical solution of this embodiment is adopted, and the gesture recognition system includes a data acquisition module, a data processing module, a fingerprint recognition module and a gesture recognition module for acquiring first video data. The acquired first video data is processed, segmented and extracted to obtain a first hand area, and then the first gesture recognition model is determined according to the first fingerprint image information using the correspondence between the fingerprint and the gesture recognition model, and then the first gesture recognition model is used to perform gesture recognition on the first hand area to obtain a first gesture recognition result, which not only recognizes gestures quickly, but also ensures a high standard recognition rate.

[0100] It should be known that Figure 1 The block diagram of the gesture recognition system based on VR glasses is shown for illustration only, and the number of modules shown therein does not limit the protection scope of the present invention.

[0101] In some possible implementations of the present invention, in the step of processing the first video image sequence, segmenting the background and the human body, and obtaining the first segmented image, the data processing module is specifically configured as follows:

[0102] Extracting current pixel points from the first video image sequence in a preset order;

[0103] Compare the pixel value of the current pixel with all the background sample pixel values ​​in the background pixel library;

[0104] Determine a first number of background sample pixel values ​​whose difference with the pixel value of the current pixel point is within a first preset range;

[0105] If the first number is greater than a first preset threshold, determining that the current pixel point belongs to a background point;

[0106] Repeat the above steps until the first video image sequence is compared and a current background point set is obtained;

[0107] Determining a background image region in the first video image sequence according to the current background point set;

[0108] The background image area is extracted from the first video image sequence to segment the background and the human body, thereby obtaining the first segmented image.

[0109] It can be understood that in order to accurately and efficiently identify the hand from the image, an embodiment of the present invention adopts a method of first deleting the image background, by extracting the current pixel point, and comparing the pixel value of the current pixel point with all background sample pixel values. When the first number of background sample pixel values ​​whose difference with the pixel value of the current pixel point is within a first preset range is greater than a first preset threshold, it is determined that the current pixel point belongs to the background point; the background points are found to form a current background point set, thereby obtaining the background image area, and the background image area is deleted to obtain a first segmented image that only retains the human body area.

[0110] In some possible implementations of the present invention, after the step of determining a connected domain in the first segmented image and marking the connected domain and before the step of determining the first hand region from the connected domain according to a preset hand region size range value, the data processing module is configured as follows:

[0111] Compressing the first segmented image into an image with a pixel value of 64*64 and performing grayscale processing;

[0112] Input the processed image into the trained convolutional neural network model to obtain the face image area;

[0113] The face image area is deleted from the connected domain.

[0114] It is understandable that in practical applications, the size of each first segmented image is different. For the convenience of subsequent operations, the first segmented image needs to be padded into a square, compressed into a pixel value of 64*64, and grayed out.

[0115] In an embodiment of the present invention, the steps of training a convolutional neural network model include: the first step is to load the data into the memory, that is, to load the image set after image preprocessing into the memory in the form of a multidimensional array, and to annotate the label value for each type of sample data; the second step is to divide the data set, that is, to divide the data set, the validation set, and the training set according to the principle of cross-validation. The third step is to change the dimension of the image according to the input order of the convolutional neural network; the fourth step is to normalize the image data, that is, the purpose of making the data set float and then normalized is to improve the network convergence speed, reduce the training practice of the model, and adapt to the activation function with a value range between (0,1) to increase the discrimination; one particularly important reason for normalization is to ensure that the eigenvalue weights are consistent; the fifth step is to determine the optimizer; the sixth step is to build a convolutional neural network, and train the convolutional neural network to obtain a convolutional neural network model.

[0116] By inputting the processed image into a trained convolutional neural network model to obtain a face image region, and deleting the face image region from the connected domain, interference from other factors can be eliminated, thereby improving the efficiency and accuracy of gesture recognition.

[0117] In some possible implementations of the present invention, in the step of extracting the first fingerprint image information from the first hand area, the fingerprint recognition module is specifically configured as follows:

[0118] extracting a first finger image from the first hand region;

[0119] Detecting and extracting a first fingerprint image from the first finger image;

[0120] Performing grayscale and normalization processing on the first fingerprint image to obtain a second fingerprint image;

[0121] Binarizing the second fingerprint image to obtain a third fingerprint image;

[0122] Performing enhancement and refinement processing on the third fingerprint image to obtain a fourth fingerprint image;

[0123] Extracting feature points from the fourth fingerprint image;

[0124] Matching the feature points with the feature points of fingerprint samples in a pre-stored fingerprint database;

[0125] When the matching value reaches the preset matching value, the information of the corresponding fingerprint sample is determined to be the first fingerprint image information.

[0126] It is understandable that in order to extract fingerprint information more accurately, in an embodiment of the present invention, a first finger image can be extracted according to a pre-trained finger recognition model, and then the first fingerprint image can be detected and recognized from the first finger image; and then the first fingerprint image is grayed and normalized to obtain a second fingerprint image. Among them, the gray processing is to simplify the image information. After a color image is grayed, only one gray value remains after gray processing, and the color information is removed; the reason for normalizing the first fingerprint image is that the ridges and valleys in the image of the collected fingerprint image will change greatly due to light, angle and other reasons, and errors are prone to occur. Then, the second fingerprint image is binarized to obtain a third fingerprint image, and the second fingerprint image is converted into a third fingerprint image with only two pixel values ​​of 0 and 1. Specifically, all the pixels in the second fingerprint image that are not in the ridge area are set to 0, and all the pixels in the part where the ridge is located are set to 1. Then, the third fingerprint image is enhanced to separate the adhered lines, smooth the edges of the lines, etc., to ensure the reliability of fingerprint feature extraction. Then, refinement processing is performed to extract the framework of the ridges in the fingerprint image, highlight the effective features of the ridges, and obtain the fourth fingerprint image to prepare for the subsequent feature extraction.

[0127] In an embodiment of the present invention, the method for extracting feature points from the fourth fingerprint image may be: first, a 3×3 template is established according to the fourth fingerprint image, the center P is the pixel to be tested, and P1 to P8 are 8 neighboring pixels of the pixel to be tested P, and CnF is the number of times the grayscale values ​​of the 8 neighboring pixels change from 0 to 1 or from 1 to 0, and the calculation formula is shown below:

[0128] Among them, P9=P1.

[0129] SnF is the number of pixels with a gray value of 1 among the 8 neighboring pixels. The calculation formula is as follows:

[0130]

[0131] By calculating the size of CnF and SnF, the state of pixel point P is analyzed, which is generally divided into the following cases: (1) If pixel point P is a point on the ridge line, and according to the calculation of CnF=2 and SnF=1, it can be known that pixel point P is an endpoint on the ridge line; (2) If pixel point P is a point on the ridge line, and according to the calculation of CnF=4 and SnF=1, 2 or 4, it can be known that pixel point P is a continuous point on the ridge line; (3) If pixel point P is a point on the ridge line, and according to the calculation of CnF=6 and SnF=3, it can be known that pixel point P is a bifurcation point on the ridge line.

[0132] The fourth fingerprint image is detected and processed, and all detected endpoints and bifurcation points are calculated according to the above method as the result of feature point extraction. The feature points are matched with the feature points of the fingerprint samples in the pre-stored fingerprint database; when the matching value reaches the preset matching value, the information of the corresponding fingerprint sample is determined to be the first fingerprint image information.

[0133] In some possible implementations of the present invention, a control module is further included; after the step of performing gesture recognition on the first hand area using the first gesture recognition model to obtain a first gesture recognition result, the control module is configured to:

[0134] generating a first control instruction according to the first recognition result;

[0135] Sending the first control instruction to the terminal to be controlled;

[0136] Control and display the execution status of the first control instruction executed by the terminal to be controlled.

[0137] It can be understood that the VR glasses in the embodiment of the present invention may further include a wireless communication module. After acquiring the user's gesture action and identifying the meaning represented by the gesture action, and obtaining a first gesture recognition result, the control module generates a first control instruction according to the first recognition result, and sends the first control instruction to the terminal to be controlled (the terminal to be controlled can be a drone, a robot, a computer, etc., or it can be the VR glasses themselves). After the terminal to be controlled receives the first control instruction and executes it, the control module controls and displays the execution status of the terminal to be controlled executing the first control instruction, so that the user can adjust the control gesture according to the execution status, so as to achieve the purpose of arbitrarily and accurately controlling the terminal to be controlled.

[0138] See also Figure 2 Another embodiment of the present invention provides a gesture recognition method based on VR glasses, comprising:

[0139] Acquire background image data and establish a background pixel library containing background sample pixel values;

[0140] Acquire first video data;

[0141] extracting a first video image sequence from the first video data;

[0142] Processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image;

[0143] Determine a connected domain in the first segmented image and mark the connected domain;

[0144] Determining a first hand region from the connected domain according to a preset hand region size range value;

[0145] Extracting first fingerprint image information from the first hand area;

[0146] determining a first gesture recognition model according to the first fingerprint image information;

[0147] Performing gesture recognition on the first hand region using the first gesture recognition model to obtain a first gesture recognition result.

[0148] It can be understood that the VR glasses in the embodiment of the present invention may include a data acquisition module, such as a camera. There may be multiple cameras. By using multiple cameras, better image data with more parameter information can be obtained; the data acquisition module may also include a touch screen, etc.

[0149] The VR glasses data processing module in the embodiment of the present invention may be a processing chip, such as a CPU, a GPU, etc.

[0150] The VR glasses in the embodiment of the present invention may further include a fingerprint recognition module, which is used to extract fingerprint data from the acquired image data and perform fingerprint recognition.

[0151] The VR glasses in the embodiment of the present invention may further include a gesture recognition module, which uses the correspondence between fingerprints and gesture recognition models to determine a first gesture recognition model according to the first fingerprint image information, and then uses the first gesture recognition model to perform gesture recognition on the first hand area to obtain a first gesture recognition result.

[0152] The technical solution of this embodiment is adopted, and the gesture recognition system includes a data acquisition module, a data processing module, a fingerprint recognition module and a gesture recognition module for acquiring first video data. The acquired first video data is processed, segmented and extracted to obtain a first hand area, and then the first gesture recognition model is determined according to the first fingerprint image information using the correspondence between the fingerprint and the gesture recognition model, and then the first gesture recognition model is used to perform gesture recognition on the first hand area to obtain a first gesture recognition result, which not only recognizes gestures quickly, but also ensures a high standard recognition rate.

[0153] In some possible implementations of the present invention, the step of processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image includes:

[0154] Extracting current pixel points from the first video image sequence in a preset order;

[0155] Compare the pixel value of the current pixel with all the background sample pixel values ​​in the background pixel library;

[0156] Determine a first number of background sample pixel values ​​whose difference with the pixel value of the current pixel point is within a first preset range;

[0157] If the first number is greater than a first preset threshold, determining that the current pixel point belongs to a background point;

[0158] Repeat the above steps until the first video image sequence is compared and a current background point set is obtained;

[0159] Determining a background image region in the first video image sequence according to the current background point set;

[0160] The background image area is extracted from the first video image sequence to segment the background and the human body, thereby obtaining the first segmented image.

[0161] It can be understood that in order to accurately and efficiently identify the hand from the image, an embodiment of the present invention adopts a method of first deleting the image background, by extracting the current pixel point, and comparing the pixel value of the current pixel point with all background sample pixel values. When the first number of background sample pixel values ​​whose difference with the pixel value of the current pixel point is within a first preset range is greater than a first preset threshold, it is determined that the current pixel point belongs to the background point; the background points are found to form a current background point set, thereby obtaining the background image area, and the background image area is deleted to obtain a first segmented image that only retains the human body area.

[0162] In some possible implementations of the present invention, after the step of determining a connected domain in the first segmented image and marking the connected domain and before the step of determining a first hand region from the connected domain according to a preset hand region size range value, the method includes:

[0163] Compressing the first segmented image into an image with a pixel value of 64*64 and performing grayscale processing;

[0164] Input the processed image into the trained convolutional neural network model to obtain the face image area;

[0165] The face image area is deleted from the connected domain.

[0166] It is understandable that in practical applications, the size of each first segmented image is different. For the convenience of subsequent operations, the first segmented image needs to be padded into a square, compressed into a pixel value of 64*64, and grayed out.

[0167] In an embodiment of the present invention, the steps of training a convolutional neural network model include: the first step is to load the data into the memory, that is, to load the image set after image preprocessing into the memory in the form of a multidimensional array, and to annotate the label value for each type of sample data; the second step is to divide the data set, that is, to divide the data set, the validation set, and the training set according to the principle of cross-validation. The third step is to change the dimension of the image according to the input order of the convolutional neural network; the fourth step is to normalize the image data, that is, the purpose of making the data set float and then normalized is to improve the network convergence speed, reduce the training practice of the model, and adapt to the activation function with a value range between (0,1) to increase the discrimination; one particularly important reason for normalization is to ensure that the eigenvalue weights are consistent; the fifth step is to determine the optimizer; the sixth step is to build a convolutional neural network, and train the convolutional neural network to obtain a convolutional neural network model.

[0168] By inputting the processed image into a trained convolutional neural network model to obtain a face image region, and deleting the face image region from the connected domain, interference from other factors can be eliminated, thereby improving the efficiency and accuracy of gesture recognition.

[0169] In some possible implementations of the present invention, the step of extracting first fingerprint image information from the first hand area includes:

[0170] extracting a first finger image from the first hand region;

[0171] Detecting and extracting a first fingerprint image from the first finger image;

[0172] Performing grayscale and normalization processing on the first fingerprint image to obtain a second fingerprint image;

[0173] Binarizing the second fingerprint image to obtain a third fingerprint image;

[0174] Performing enhancement and refinement processing on the third fingerprint image to obtain a fourth fingerprint image;

[0175] Extracting feature points from the fourth fingerprint image;

[0176] Matching the feature points with the feature points of fingerprint samples in a pre-stored fingerprint database;

[0177] When the matching value reaches the preset matching value, the information of the corresponding fingerprint sample is determined to be the first fingerprint image information.

[0178] It is understandable that in order to extract fingerprint information more accurately, in an embodiment of the present invention, a first finger image can be extracted according to a pre-trained finger recognition model, and then the first fingerprint image can be detected and recognized from the first finger image; and then the first fingerprint image is grayed and normalized to obtain a second fingerprint image. Among them, the gray processing is to simplify the image information. After a color image is grayed, only one gray value remains after gray processing, and the color information is removed; the reason for normalizing the first fingerprint image is that the ridges and valleys in the image of the collected fingerprint image will change greatly due to light, angle and other reasons, and errors are prone to occur. Then, the second fingerprint image is binarized to obtain a third fingerprint image, and the second fingerprint image is converted into a third fingerprint image with only two pixel values ​​of 0 and 1. Specifically, all the pixels in the second fingerprint image that are not in the ridge area are set to 0, and all the pixels in the part where the ridge is located are set to 1. Then, the third fingerprint image is enhanced to separate the adhered lines, smooth the edges of the lines, etc., to ensure the reliability of fingerprint feature extraction. Then, refinement processing is performed to extract the framework of the ridges in the fingerprint image, highlight the effective features of the ridges, and obtain the fourth fingerprint image to prepare for the subsequent feature extraction.

[0179] In an embodiment of the present invention, the method for extracting feature points from the fourth fingerprint image may be: first, a 3×3 template is established according to the fourth fingerprint image, the center P is the pixel to be tested, and P1 to P8 are 8 neighboring pixels of the pixel to be tested P, and CnF is the number of times the grayscale values ​​of the 8 neighboring pixels change from 0 to 1 or from 1 to 0, and the calculation formula is shown below:

[0180] Among them, P9=P1.

[0181] SnF is the number of pixels with a gray value of 1 among the 8 neighboring pixels. The calculation formula is as follows:

[0182]

[0183] By calculating the size of CnF and SnF, the state of pixel point P is analyzed, which is generally divided into the following cases: (1) If pixel point P is a point on the ridge line, and according to the calculation of CnF=2 and SnF=1, it can be known that pixel point P is an endpoint on the ridge line; (2) If pixel point P is a point on the ridge line, and according to the calculation of CnF=4 and SnF=1, 2 or 4, it can be known that pixel point P is a continuous point on the ridge line; (3) If pixel point P is a point on the ridge line, and according to the calculation of CnF=6 and SnF=3, it can be known that pixel point P is a bifurcation point on the ridge line.

[0184] The fourth fingerprint image is detected and processed, and all detected endpoints and bifurcation points are calculated according to the above method as the result of feature point extraction. The feature points are matched with the feature points of the fingerprint samples in the pre-stored fingerprint database; when the matching value reaches the preset matching value, the information of the corresponding fingerprint sample is determined to be the first fingerprint image information.

[0185] In some possible implementations of the present invention, after the step of performing gesture recognition on the first hand region using the first gesture recognition model to obtain a first gesture recognition result, the following steps are included:

[0186] generating a first control instruction according to the first recognition result;

[0187] Sending the first control instruction to the terminal to be controlled;

[0188] The execution status of the first control instruction executed by the terminal to be controlled is displayed.

[0189] It can be understood that the VR glasses in the embodiment of the present invention may further include a wireless communication module. After acquiring the user's gesture action and identifying the meaning represented by the gesture action, and obtaining a first gesture recognition result, a first control instruction is generated according to the first recognition result, and the first control instruction is sent to the terminal to be controlled (the terminal to be controlled can be a drone, a robot, a computer, etc., or it can be the VR glasses themselves). After the terminal to be controlled receives the first control instruction and executes it, the execution status of the first control instruction executed by the terminal to be controlled is controlled and displayed, so that the user can adjust the control gesture according to the execution status, so as to achieve the purpose of arbitrarily and accurately controlling the terminal to be controlled.

[0190] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0191] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0192] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0193] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0195] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0196] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0197] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0198] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.

Claims

1. A gesture recognition system based on VR glasses, characterized in that: include: A data acquisition module, a data processing module, a fingerprint recognition module and a gesture recognition module for acquiring first video data; wherein: The data acquisition module is configured to: acquire background image data and establish a background pixel library containing background sample pixel values; The data processing module is configured as follows: extracting a first video image sequence from the first video data; Processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image; Determine a connected domain in the first segmented image and mark the connected domain; Determining a first hand region from the connected domain according to a preset hand region size range value; The fingerprint recognition module is configured to: extract first fingerprint image information from the first hand area; The gesture recognition module is configured as follows: Determine a first gesture recognition model according to the first fingerprint image information by using the correspondence between the fingerprint and the gesture recognition model; Using the first gesture recognition model to perform gesture recognition on the first hand area to obtain a first gesture recognition result; Wherein, in the step of processing the first video image sequence, segmenting the background and the human body, and obtaining the first segmented image, the data processing module is specifically configured as follows: Extracting current pixel points from the first video image sequence in a preset order; Compare the pixel value of the current pixel with all the background sample pixel values ​​in the background pixel library; Determine a first number of background sample pixel values ​​whose difference with the pixel value of the current pixel point is within a first preset range; If the first number is greater than a first preset threshold, determining that the current pixel point belongs to a background point; Repeat the above steps until the first video image sequence is compared and a current background point set is obtained; Determining a background image region in the first video image sequence according to the current background point set; Extracting the background image region from the first video image sequence to segment the background and the human body, thereby obtaining the first segmented image; Wherein, after the step of determining a connected domain in the first segmented image and marking the connected domain and before the step of determining the first hand region from the connected domain according to a preset hand region size range value, the data processing module is configured as follows: Compressing the first segmented image into an image with a pixel value of 64*64 and performing grayscale processing; Input the processed image into the trained convolutional neural network model to obtain the face image area; Deleting the face image area from the connected domain; Wherein, in the step of extracting the first fingerprint image information from the first hand area, the fingerprint recognition module is specifically configured as follows: extracting a first finger image from the first hand region; Detecting and extracting a first fingerprint image from the first finger image; Performing grayscale and normalization processing on the first fingerprint image to obtain a second fingerprint image; Binarizing the second fingerprint image to obtain a third fingerprint image; Performing enhancement and refinement processing on the third fingerprint image to obtain a fourth fingerprint image; Extracting feature points from the fourth fingerprint image; Matching the feature points with the feature points of fingerprint samples in a pre-stored fingerprint database; When the matching value reaches the preset matching value, the information of the corresponding fingerprint sample is determined to be the first fingerprint image information.

2. The gesture recognition system based on VR glasses according to claim 1, characterized in that: It also includes a control module; after the step of using the first gesture recognition model to perform gesture recognition on the first hand area to obtain a first gesture recognition result, the control module is configured to: generating a first control instruction according to the first recognition result; Sending the first control instruction to the terminal to be controlled; Control and display the execution status of the first control instruction executed by the terminal to be controlled.

3. A gesture recognition method based on VR glasses, characterized in that: include: Acquire background image data and establish a background pixel library containing background sample pixel values; Acquire first video data; extracting a first video image sequence from the first video data; Processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image; Determine a connected domain in the first segmented image and mark the connected domain; Determining a first hand region from the connected domain according to a preset hand region size range value; Extracting first fingerprint image information from the first hand area; Determine a first gesture recognition model according to the first fingerprint image information by using the correspondence between the fingerprint and the gesture recognition model; Using the first gesture recognition model to perform gesture recognition on the first hand area to obtain a first gesture recognition result; The step of processing the first video image sequence, segmenting the background and the human body, and obtaining a first segmented image includes: Extracting current pixel points from the first video image sequence in a preset order; Compare the pixel value of the current pixel with all the background sample pixel values ​​in the background pixel library; Determine a first number of background sample pixel values ​​whose difference with the pixel value of the current pixel point is within a first preset range; If the first number is greater than a first preset threshold, determining that the current pixel point belongs to a background point; Repeat the above steps until the first video image sequence is compared and a current background point set is obtained; Determining a background image region in the first video image sequence according to the current background point set; Extracting the background image region from the first video image sequence to segment the background and the human body, thereby obtaining the first segmented image; Wherein, after the step of determining a connected domain in the first segmented image and marking the connected domain and before the step of determining a first hand region from the connected domain according to a preset hand region size range value, the method further includes: Compressing the first segmented image into an image with a pixel value of 64*64 and performing grayscale processing; Input the processed image into the trained convolutional neural network model to obtain the face image area; Deleting the face image area from the connected domain; The step of extracting first fingerprint image information from the first hand area includes: extracting a first finger image from the first hand region; Detecting and extracting a first fingerprint image from the first finger image; Performing grayscale and normalization processing on the first fingerprint image to obtain a second fingerprint image; Binarizing the second fingerprint image to obtain a third fingerprint image; Performing enhancement and refinement processing on the third fingerprint image to obtain a fourth fingerprint image; Extracting feature points from the fourth fingerprint image; Matching the feature points with the feature points of fingerprint samples in a pre-stored fingerprint database; When the matching value reaches the preset matching value, the information of the corresponding fingerprint sample is determined to be the first fingerprint image information.

4. The gesture recognition method based on VR glasses according to claim 3 is characterized in that: After the step of performing gesture recognition on the first hand area using the first gesture recognition model to obtain a first gesture recognition result, the method further includes: generating a first control instruction according to the first recognition result; Sending the first control instruction to the terminal to be controlled; The execution status of the first control instruction executed by the terminal to be controlled is displayed.

Images