Thermal imaging gesture recognition method based on binary neural network

Abstract

The present invention provides a thermal imaging gesture recognition method based on a binary neural network, which includes: collecting data of different gesture actions through an infrared thermal imaging sensor, visualizing and integrating the collected data, training the gesture data set using a binary neural network model, inputting the gesture to be recognized, and obtaining the recognition result. By combining gesture recognition with an infrared thermal imager, the method enables the human-computer interaction of gesture recognition to be applicable to various complex environments, avoids interference due to complex backgrounds or lighting, ensures simple and lightweight operations, and improves the accuracy and efficiency of recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a thermal imaging gesture recognition method based on a binary neural network. Background Art

[0002] At present, human-computer interaction technology usually uses computer vision to identify and obtain data, so as to achieve the purpose of becoming the entrance of human-computer interaction; compared with face recognition, gait recognition, facial tag recognition and body recognition methods, gesture recognition has gradually become a key technology and research focus in the field of human-computer interaction because of its natural, intuitive, rich and more in line with human habits. Gesture recognition is mainly achieved through two methods: machine vision cameras and wearable sensors: machine vision cameras generally collect images by taking pictures or videos to obtain image fragments at a certain moment, and then process the collected images through computers to achieve the purpose of completing human-computer interaction according to gesture features and / or definitions; wearable sensors usually collect and process data through sensors integrated into wearable devices, and then identify and match with defined gesture templates to achieve the purpose of human-computer interaction. However, when a machine vision camera takes pictures or videos of gestures, it is greatly affected by lighting and background and has high limitations. The accuracy of recognition is greatly affected by the clarity of the image or video. There are many factors that may cause errors with a high probability. The recognition cycle is long and the efficiency is low. The recognition accuracy of wearable sensors is greatly affected by the sensitivity of the sensors. In addition, the production cost is high, the amount of data acquired is huge, it is not easy to popularize, and the scope of application is low. Summary of the invention

[0003] In view of the problems existing in the above prior art, the purpose of the present invention is to provide a thermal imaging gesture recognition method based on a binary neural network. The method combines gesture recognition with an infrared thermal imager so that the human-computer interaction of gesture recognition can be applied to various complex environments, effectively avoiding the influence of lighting environment, sensor equipment, etc., thereby improving the accuracy and efficiency of recognition.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] A thermal imaging gesture recognition method based on a binary neural network, characterized by comprising:

[0006] Gesture data set collection: collect data of different gestures through infrared thermal imaging sensors (i.e. collect a variety of different gesture data);

[0007] Data preprocessing: First, the infrared thermal images collected by the infrared heat transfer sensor are visualized to obtain the grayscale image data of the gesture, thereby solving the problem of low brightness and low contrast of the grayscale image data; then, the grayscale image data of all gestures are integrated using similarity to form a gesture data set;

[0008] Gesture recognition model training: Split the gesture data set into training set, validation set and test set according to a certain ratio, and input the training set into the binary neural network model for training. After training, use the validation set for optimization to obtain the gesture recognition model.

[0009] Gesture recognition: The gesture to be recognized is input into the gesture recognition model through the infrared thermal imaging sensor to perform gesture prediction and obtain the gesture recognition result.

[0010] Based on the further optimization of the above scheme, the infrared thermal imaging sensor includes an optical imaging objective lens, an optical mechanical scanning system and an infrared detector;

[0011] The method of collecting gesture movements with infrared thermal imaging is as follows: first, an infrared image of the object to be detected is taken through an optical imaging objective lens; then, the infrared thermal image of the object to be detected is scanned by an optical scanning system and focused on an infrared detector, which converts the radiation of the infrared thermal image into an electrical signal, amplifies the electrical signal, and performs analog-to-digital conversion with a microcontroller, finally obtaining an infrared thermal image.

[0012] Based on the further optimization of the above scheme, the method of visualization processing in the data preprocessing is specifically as follows:

[0013] First, the acquired infrared thermal image is decomposed into several pixels, and the pixels are processed as follows:

[0014]

[0015] In the formula, gray represents the pixel vector after processing, and t represents the pixel vector before processing;

[0016] Then, preset the high temperature threshold T h With low temperature threshold T l , the temperature of the pixel is mapped by the high temperature threshold and the low temperature threshold, as follows:

[0017]

[0018] Where: V l Indicates low pixel value, V h Indicates high pixel value, and V l With V h All are preset values; i represents the number of pixels on the infrared thermal image;

[0019] Finally, all processed pixels are spliced ​​together to obtain the grayscale image data of the gesture.

[0020] During the room temperature test acquisition process, the values ​​obtained are about 15-25°C; when a human body passes by, due to the distance, the human body is wrapped in clothes, and the sensor is not temperature calibrated, the tested human body temperature will not reach normal body temperature, generally around 30°C. If the collected temperature value is directly used as an image, since the temperature values ​​are concentrated in a narrow range in most environments and the entire grayscale space is not used, the collected grayscale image will show low brightness and low contrast, and the content of the image cannot be accurately and clearly observed, which will affect the subsequent training and test results. Therefore, through data preprocessing, that is, through the setting of the temperature mapping threshold interval, the low-brightness and low-contrast image is transformed into a high-contrast image, thereby obtaining a clear image, increasing the accuracy of training and testing, and reducing errors.

[0021] Based on the further optimization of the above scheme, the specific method of data preprocessing and integration processing is as follows:

[0022] First, using Euclidean distance as the similarity measure, all pixels of the grayscale images of two different gestures in the gesture dataset are compared for similarity, specifically:

[0023]

[0024] Where: represents the pixel coordinates of the first gesture grayscale image data, Represents pixel coordinates of the second gesture grayscale image data;

[0025] Then, get the average value of the distance difference between the two gesture grayscale images:

[0026]

[0027] After that, the similarity threshold D is preset: if d 0 ≤D, the two gesture grayscale image data are similar and are merged into the same gesture data subset; if d 0 >D, the two gesture grayscale image data are not similar, and the two gesture grayscale image data are compared with other gesture grayscale image data respectively;

[0028] Finally, a gesture data set is formed that includes different gesture data subsets (such as a rock gesture data subset, a scissors gesture data subset, a paper gesture data subset, an OK gesture data subset, a Good gesture data subset, etc.).

[0029] Based on further optimization of the above scheme, the ratio of the training set, validation set and test set is 7:2:1.

[0030] Based on the further optimization of the above scheme, the binary neural network model consists of a convolution layer, a batch normalization layer, a binary activation layer, a convolution layer, a batch normalization layer, a maximum pooling layer, and a binary activation layer to form a feature extraction block, and a fully connected layer, a batch normalization layer, a fully connected layer, and a binary activation layer to form a feature classification block; wherein the binary neural network model repeats T 1 Second feature extraction block, T 2 Secondary feature classification block, and the softmax output layer is used as the model category output;

[0031] The loss function of the binary neural network model is:

[0032]

[0033] Where: y t represents the estimated vector, y p Represents the true vector; N represents the vector length, and r represents the loss function result.

[0034] Based on further optimization of the above solution, in the gesture recognition, before the gesture to be recognized is input into the gesture recognition model through the infrared thermal imaging sensor, it is necessary to perform visualization processing on the data input by the infrared thermal imaging sensor to obtain the grayscale image data of the gesture.

[0035] The present invention has the following technical effects:

[0036] The present application collects gesture data through infrared thermal imaging, and uses a binary neural network model to train and recognize gesture data. Compared with the traditional convolutional neural network structure with huge computational complexity, the present application method can efficiently and accurately complete gesture recognition with high recognition reliability and low error rate without any loss in model performance. At the same time, the present application method uses infrared thermal imaging to detect gestures, which effectively solves the influence of complex background or illumination during imaging, avoids the interference of uncontrollable environmental factors during recognition, and further improves the accuracy of recognition. After testing, the recognition rate of the present application method for the three gestures of "rock, scissors, cloth" is 99.3%, and the recognition rate of the five gestures of "rock, scissors, cloth, OK, Good" is 95.1%; in addition, the present application method achieves inference and recognition speeds of 2.7ms and 22.7ms in three gesture data sets and five gesture data sets, respectively, which effectively shortens the duration of gesture recognition, reduces time cost, and accelerates training effect. It can also be applied to gesture recognition in various environments and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of an infrared thermal imaging sensor obtaining gesture data in an embodiment of the present invention.

[0038] Figure 2 It is a comparison diagram before and after the visualization processing in the embodiment of the present invention; wherein, Figure 2 (a) is a schematic diagram before visualization. Figure 2 (b) is the schematic diagram after visualization processing.

[0039] Figure 3 It is a gesture data set formed after integration processing in an embodiment of the present invention.

[0040] Among them, 10, the object to be detected; 20, the optical imaging objective lens; 30, the optical mechanical scanning system; 40, the infrared detector. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0042] Embodiment 1:

[0043] like Figures 1 to 3 As shown, a thermal imaging gesture recognition method based on a binary neural network is characterized by comprising:

[0044] Gesture data set collection: collect data of different gestures through infrared thermal imaging sensors (i.e. collect a variety of different gesture data);

[0045] like Figure 1 As shown: the infrared thermal imaging sensor includes an optical imaging objective lens 20, an optical scanning system 30 and an infrared detector 40; its method for collecting gesture actions is: first, infrared imaging of the object to be detected 10 is performed through the optical imaging objective lens 20; then, the infrared thermal image of the object to be detected 10 is scanned by the optical scanning system 30 and focused on the infrared detector 40, and the infrared detector 40 converts the radiation of the infrared thermal image into an electrical signal, and the electrical signal is amplified and processed and converted into analog-to-digital by the microcontroller, and finally an infrared thermal image is obtained.

[0046] In this embodiment, the infrared thermal imaging sensor model used is MLX90640, which supports the output of 32*24 pixel array, a total of 768 FIR (Far Infra-Red) pixels, supports I2C protocol up to 1MHz, has a small viewing angle, can accurately detect targets in specific areas and temperature ranges, and meets the requirements of long-distance temperature measurement.

[0047] Data preprocessing: First, the infrared thermal images collected by the infrared heat transfer sensor are visualized to obtain the grayscale image data of the gesture, thereby solving the problem of low brightness and low contrast of the grayscale image data. The specific steps are as follows:

[0048] First, the acquired infrared thermal image (i.e. Figure 2 The image (a) is decomposed into several pixels, and the pixels are processed as follows:

[0049]

[0050] In the formula, gray represents the pixel vector after processing, and t represents the pixel vector before processing;

[0051] Then, preset the high temperature threshold T h With low temperature threshold T l , the temperature of the pixel is mapped by the high temperature threshold and the low temperature threshold, as follows:

[0052]

[0053] Where: V l Indicates low pixel value, V h Indicates high pixel value, and V l With V h All are preset values; i represents the number of pixels on the infrared thermal image;

[0054] In this embodiment, T l The preset value is 10℃, T h The default setting is 40℃; V l The default setting is 0, V h The default value is 0;

[0055] Finally, all the processed pixels are spliced ​​to obtain the grayscale image data of the gesture (i.e. Figure 2 (b) shows the image).

[0056] Afterwards, the grayscale image data of all gestures are integrated using similarity to form a gesture dataset. The specific steps are as follows:

[0057] First, using Euclidean distance as the similarity measure, all pixels of the grayscale images of two different gestures in the gesture dataset are compared for similarity, specifically:

[0058]

[0059] Where: represents the pixel coordinates of the first gesture grayscale image data, Represents the pixel coordinates of the second gesture grayscale image data;

[0060] Then, get the average value of the distance difference between the two gesture grayscale images:

[0061]

[0062] After that, a similarity threshold D is preset (the similarity threshold D is determined based on a large amount of empirical data actually obtained): 0 ≤D, the two gesture grayscale image data are similar and are merged into the same gesture data subset; if d 0 >D, the two gesture grayscale image data are not similar, and the two gesture grayscale image data are compared with other gesture grayscale image data respectively;

[0063] Finally, a gesture data set is formed that includes different gesture data subsets (such as a rock gesture data subset, a scissors gesture data subset, a paper gesture data subset, an OK gesture data subset, a Good gesture data subset, etc.); that is, the gesture data set is divided into several subsets such as a rock gesture data subset, a scissors gesture data subset, a paper gesture data subset, an OK gesture data subset, and a Good gesture data subset, and the most representative gesture grayscale image data is selected as the first gesture grayscale image data of the corresponding subset, and the similarity of each detected second gesture grayscale image data and the first gesture grayscale image data is compared with the similarity of the above method respectively. If they are similar, the second gesture grayscale image data is stored in the corresponding gesture data subset. If they are not similar, the second gesture grayscale image data is compared with the first gesture grayscale image data of other gesture data subsets until the gesture data subset corresponding to the second gesture grayscale image data is found; each gesture grayscale image data is traversed to complete the establishment of the gesture data subset, and finally the gesture data set is obtained. Figure 3 shown.

[0064] Gesture recognition model training: The gesture data set is split into a training set, a validation set, and a test set in a certain ratio. In this embodiment, the ratio of the training set, the validation set, and the test set is 7:2:1.

[0065] The training set is input into the binary neural network model for training, and after training, the validation set is used for optimization to obtain the gesture recognition model;

[0066] The binary neural network model consists of a convolutional layer, a batch normalization layer, a binary activation layer, a convolutional layer, a batch normalization layer, a maximum pooling layer, and a binary activation layer to form a feature extraction block, and a fully connected layer, a batch normalization layer, a fully connected layer, and a binary activation layer to form a feature classification block; the binary neural network model repeats T 1 Second feature extraction block, T 2 Secondary feature classification block, in this embodiment: T 1 Set to 2, T 2 Set to 3, and use the softmax output layer as the model category output;

[0067] The traditional convolutional neural network model structure includes convolution layer, batch normalization layer, activation layer, pooling layer, and full connection; the core of the convolutional neural network is to use the convolution operation to obtain the convolution kernel features that minimize the loss function, so that the feature parameters of the network model are much less than those of the multi-layer perceptron, and solve the problem that the multi-layer perceptron model is too deep and cannot be trained; the convolution operation is particularly important, and the two-dimensional discrete convolution formula is:

[0068]

[0069] In the formula, f(m,n) and g(m,n) represent the functions of the convolution operation, o(m,n) represents the result of the convolution operation; m and n represent the index of the current convolution operation result;

[0070] In a convolutional neural network, the convolution operation is to convolve each input feature map with a convolution kernel of size s×s, sum the results of the convolution operation, add the bias, and finally activate the output feature map through a nonlinear function f(x) (such as Sigmoid, ReLU, Tanh); the convolutional neural network formula is:

[0071]

[0072] Where: x i,j Represents the pixel at position (i, j) of the feature map of the input image; k p,q Represents the element value of the convolution kernel at position (p, q); b represents the bias of the convolution; O(i, j) represents the result obtained by the convolution, and i and j represent the index of the current feature map position;

[0073] The binary neural network model is to binarize the weights of the convolutional layer and the fully connected layer in the convolutional neural network and the output of the activation layer. Specifically:

[0074]

[0075] Where: w b is the result of binary quantization of w, where w represents each parameter in the weight;

[0076] The convolution operation in the binary neural network model of this embodiment can be replaced by bit operations such as XNOR operation and Bitcount.

[0077] The loss function of the binary neural network model is:

[0078]

[0079] Where: y t represents the estimated vector, y pRepresents the true vector; N represents the vector length, and r represents the loss function result; using classification cross entropy as the loss function can force the network to learn the distribution law of the data, thereby enabling the binary neural network model to obtain better generalization ability on random data, thereby making the result more authentic and accurate.

[0080] Gesture recognition: The gesture to be recognized is input into the gesture recognition model through the infrared thermal imaging sensor to perform gesture prediction and obtain the gesture recognition result; wherein, before the gesture to be recognized is input into the gesture recognition model through the infrared thermal imaging sensor, the data input by the infrared thermal imaging sensor needs to be visualized as described above to obtain the grayscale image data of the gesture.

[0081] Embodiment 2:

[0082] As a further optimization of the scheme of the present invention, a thermal imaging gesture recognition system based on a binary neural network is further provided in the present embodiment. Based on the scheme of the above-mentioned embodiment 1, the system uses a deep learning framework of Google's tensorflow and is trained on a GTX1650TI-4G graphics card; when the training target is a data set consisting of three gesture subsets, namely, a rock gesture data subset, a scissors gesture data subset, and a cloth gesture data subset, a bnm network model is used; when the training target is a data set consisting of five gesture subsets, namely, a rock gesture data subset, a scissors gesture data subset, a cloth gesture data subset, an OK gesture data subset, and a Good gesture data subset, a baseline network model is used.

Claims

1. A thermal imaging gesture recognition method based on binary neural network, characterized in that: include: Gesture data set collection: Collect data of different gestures through infrared thermal imaging sensors; Data preprocessing: First, the infrared thermal image acquired by the infrared heat transfer sensor is visualized to obtain the grayscale image data of the gesture; the specific visualization processing method is as follows: First, the acquired infrared thermal image is decomposed into several pixels, and the pixels are processed as follows: In the formula, gray Represents the processed pixel vector, t Represents the pixel vector before processing; Then, preset the high temperature threshold T h With low temperature threshold T l , the temperature of the pixel is mapped by the high temperature threshold and the low temperature threshold, as follows: Where: V l Indicates low pixel value, V h indicates a high pixel value, and V l and V h All are preset values; i Indicates the number of pixels on the infrared thermal image; Finally, all processed pixels are spliced ​​together to obtain the grayscale image data of the gesture; Afterwards, the grayscale image data of all gestures are integrated using similarity to form a gesture dataset; the specific method of integration is: First, using Euclidean distance as the similarity measure, all pixels of the grayscale images of two different gestures in the gesture dataset are compared for similarity, specifically: Where: represents the pixel coordinates of the first gesture grayscale image data, Represents the pixel coordinates of the second gesture grayscale image data; Then, get the average value of the distance difference between the two gesture grayscale images: After that, preset the similarity threshold D :like d 0 ≤D , then the two gesture grayscale image data are similar, and they are merged into the same gesture data subset; like d 0 ＞D , then the two gesture grayscale image data are not similar, and the two gesture grayscale image data are compared with other gesture grayscale image data respectively; Finally, a gesture dataset containing different subsets of gesture data is formed; Gesture recognition model training: Split the gesture data set into training set, validation set and test set according to a certain ratio, and input the training set into the binary neural network model for training. After training, use the validation set for optimization to obtain the gesture recognition model. The binary neural network model consists of a convolutional layer, a batch normalization layer, a binary activation layer, a convolutional layer, a batch normalization layer, a maximum pooling layer, and a binary activation layer to form a feature extraction block, and a fully connected layer, a batch normalization layer, a fully connected layer, and a binary activation layer to form a feature classification block; the binary neural network model repeats T 1 feature extraction block, T Second feature classification block, and softmax The output layer is used as the model category output; The loss function of the binary neural network model is: Where: y t represents the estimated vector, y p represents a real vector; N represents the length of the vector, r Represents the loss function result; Gesture recognition: The gesture to be recognized is input into the gesture recognition model through the infrared thermal imaging sensor to perform gesture prediction and obtain the gesture recognition result.

2. The method for thermal imaging gesture recognition based on binary neural network according to claim 1, characterized in that: The infrared thermal imaging sensor includes an optical imaging objective lens, an optical mechanical scanning system and an infrared detector; The method of collecting gesture movements with infrared thermal imaging is as follows: first, an infrared image of the object to be detected is taken through an optical imaging objective lens; then, the infrared thermal image of the object to be detected is scanned by an optical scanning system and focused on an infrared detector, which converts the radiation of the infrared thermal image into an electrical signal, amplifies the electrical signal, and performs analog-to-digital conversion with a microcontroller, finally obtaining an infrared thermal image.

3. The method for thermal imaging gesture recognition based on binary neural network according to claim 1 or 2, characterized in that: In the gesture recognition, before the gesture to be recognized is input into the gesture recognition model through the infrared thermal imaging sensor, the data input by the infrared thermal imaging sensor is visualized to obtain the grayscale image data of the gesture.

Images