Edge calculation-based infrared sensor human eye state monitoring method and system

Abstract

The invention provides an infrared sensor human eye state monitoring method and system based on edge calculation. The method comprises the following steps that thermal imaging video data of human eyes are collected in real time through an infrared sensor; performing data preprocessing on the acquired thermal imaging video data; training a convolutional neural network by using the processed data to obtain a trained neural network model; carrying out model lightweight on the trained network model by adopting a full integer quantization method; the method comprises the following steps of: deploying a human eye model on an MCU (Microprogrammed Control Unit) chip by adopting a TensorFlow Lite Micro framework, realizing human eye state real-time monitoring based on edge calculation in cooperation with an infrared sensor, and finally identifying the state of a human eye to be monitored by using the model. According to the invention, human body actions can be identified in a dark environment, and user privacy is effectively protected; an improved convolutional neural network and a strong quantization method are adopted, the weight of the model is remarkably reduced, the reasoning speed is increased, and operation on mobile equipment and embedded equipment is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and embedded system development, and in particular to a method and system for monitoring human eye status using an infrared sensor based on edge computing. Background Art

[0002] In the field of computer vision, human eye state recognition methods have become an important research direction and have attracted extensive attention from researchers at home and abroad. In recent years, with the rapid development of deep learning and computer vision technology, various human eye state recognition technologies have emerged.

[0003] At present, the main methods for human eye state recognition include using infrared sensors to capture eye thermal radiation signals and analyzing eye image features through convolutional neural networks (CNN). Both methods have their own advantages and disadvantages. Infrared sensors can work effectively in low-light environments and are suitable for scenarios with high privacy requirements, but their performance is limited by the characteristics of the sensor. They can usually only effectively identify eye movement states under specific conditions, and their real-time tracking capabilities are limited. In contrast, the CNN method performs well in a variety of environments and can extract eye features for state classification through deep learning. However, due to its complex network structure and large number of parameters, it has relatively high requirements for computing resources and is not suitable for resource-constrained embedded systems.

[0004] At the same time, most existing methods for recognizing human eye states rely on visible light imaging and cannot be effectively recognized in low-light environments. At the same time, with the increasing concern about privacy protection, the acceptance of traditional visible light imaging methods has gradually decreased. Therefore, there is an urgent need for a recognition system that can work effectively in a variety of environments, especially in low light and high privacy requirements. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention proposes a method and system for monitoring the human eye state using an infrared sensor based on edge computing.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: A method for monitoring human eye status using an infrared sensor based on edge computing comprises the following steps: S100: Collects thermal imaging video data of human eye status through infrared sensors; S200: Processing the thermal imaging video data of human eyes to obtain an eye-opening and eye-closing picture data set; S300: Train the preset TinyCNN convolutional neural network using the processed photo data set to obtain a trained model; S400: Based on the TensorFlow Lite framework, the trained model is lightweighted, the floating point type in the model calculation graph is converted into an integer type, and the model weight is reduced; S500: Use TensorFlow Lite Micro framework to deploy the model to MCU; S600: Cooperates with infrared sensors to realize real-time monitoring of human eye status based on edge computing, displays recognition prediction values ​​and updates them continuously, 0 means eyes closed, and 1 means eyes open.

[0007] Preferably, step S100 further includes: S110: Arranging the infrared sensors according to the field of view of the infrared sensors so that the human eye area is located within the field of view of the infrared sensors; S120: The infrared sensor is enabled to continuously collect frames of the human eye at a preset frame rate to obtain raw data of the human eye state.

[0008] Preferably, step S200 further includes: S210: Collecting thermal imaging video data of the human eye state at a preset time interval to obtain a plurality of photos; S220: Mark each photo with a name of the eye state, and all photos and their corresponding names form a frame data set.

[0009] Preferably, the convolutional neural network includes a first convolutional layer, a first batch normalization layer, a first pooling layer, a second convolutional layer, a second batch normalization layer, a second pooling layer, a third convolutional layer, a third batch normalization layer, a third pooling layer, a fourth convolutional layer, a fourth batch normalization layer, a fourth pooling layer, a flattening layer, a first fully connected layer, a Dropout layer and an output layer, which are connected in sequence.

[0010] Preferably, the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer are all conventional two-dimensional convolution layers with a convolution kernel size of 3×3, and the first convolution layer has 32 convolution kernels, the second convolution layer has 64 convolution kernels, and the third convolution layer and the fourth convolution layer each have 128 convolution kernels.

[0011] Preferably, the first pooling layer, the second pooling layer and the third pooling layer are all two-dimensional maximum pooling layers with a size of 2×2. This structural design aims to effectively extract the characteristics of the human eye state and optimize the learning ability of the model.

[0012] Preferably, step S300 includes: S310: Randomly take 70% of the data set obtained in S200 as a training set and 30% as a test set; S320: Use the processed data to train the pre-trained neural network model to obtain a trained neural network model.

[0013] Preferably, step S40 includes: S410: Set the converter to use integer quantization. The supported operation type is integer quantization. Both input and output are set to integer quantization.

[0014] S420: Convert the model and save the model to a file.

[0015] Preferably, step S500 includes: S510: Use the TensorFlow Lite Micro framework to deploy the model to the project.

[0016] S520: Compile the code and upload it to the MCU.

[0017] Preferably, step S600 includes: S610: The infrared camera is used to obtain the state of the human eye that needs to be monitored, and the video frames are extracted. The extraction speed can be selected.

[0018] S620: Preprocess the acquired frame image, such as converting BGR to RGB, converting numpy array to PIL image, adjusting the image size, normalizing it and then sending it to the network model for recognition.

[0019] S630: Continuously refresh the frame data obtained from the video and display the predicted value of the video frame image to achieve real-time monitoring of the human eye state based on edge computing.

[0020] Preferably, in the process of training the preset convolutional neural network, an adaptive moment estimation optimization algorithm is used to find the optimal parameters.

[0021] Another aspect of the present invention provides an infrared sensor human eye state monitoring system based on edge computing, comprising: An acquisition module, configured to acquire thermal imaging video data of human eye status through an infrared sensor; A processing module, configured to process the thermal imaging video data of the human eye state to obtain processed data; A training module is configured to train a preset convolutional neural network using the processed data to obtain a trained neural network model; Lightweight module, using the full integer method to convert the model to TensorFlow Lite type; Deploy the module, integrate the model into the project, load and run the model, compile and upload the recognition code; The recognition module is configured to recognize the eye state of the human body to be recognized by the program running on the obtained MCU, and obtain the recognition result prediction result.

[0022] Compared with the prior art, the present invention has the following beneficial effects: Thermal imaging video data of the human eye state is obtained through infrared sensors, which can be run on resource-constrained embedded systems. It can also recognize human movements in dark environments without infringing user privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flow chart of a method for monitoring human eye status using an infrared sensor based on edge computing according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a convolutional neural network of a human eye state monitoring method using an infrared sensor based on edge computing according to an embodiment of the present invention; Figure 3 2 is a schematic structural diagram of an infrared sensor human eye state monitoring system based on edge computing according to another embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0025] In nature, whether it is an organism or an inanimate object, as long as its temperature is above absolute zero (-273°C), it will generate infrared radiation. The energy of this radiation is proportional to the fourth power of the object's temperature, and its wavelength is inversely proportional to the temperature. Thermal imaging technology processes the radiation energy of the detected object and converts it into a thermal image, and then displays the temperature distribution of the target object in grayscale or pseudo-color to determine its state. Thermal imaging technology mainly records the object's own infrared radiation and converts the thermal image of the scene into a visible two-dimensional image.

[0026] like Figure 1 As shown, an embodiment of the present invention provides a method for monitoring human eye status using an infrared sensor based on edge computing, comprising the steps of: S100: Collects thermal imaging video data of human eye status through infrared sensors.

[0027] The infrared sensor can process data using infrared rays, and thermal imaging video data of the state of human eyes can be obtained through the sensor. The infrared sensor can use a specific model of sensor.

[0028] Since infrared sensors have limited field of view, the target to be detected (i.e., human eyes) must be within the effective range of the sensor to ensure the integrity of the human eye state. The installation height and position of the infrared sensor must take into account the user's visual activity range to ensure effective recognition.

[0029] Step S100 further includes: S110: Arrange sensors according to the field of view angle of the infrared sensor to ensure that the human eye activity area is within the field of view angle; S120: enabling the infrared sensor to continuously collect frames of the human eye activity area at a preset frame rate to obtain raw data; In some embodiments, the human eye state includes two states: open eyes and closed eyes. Data collection is completed by volunteers, who repeatedly perform eye movement states multiple times in the activity area so that the infrared sensor can obtain corresponding data.

[0030] In step S120 , the preset frame rate may be set to 9 FPS (frames per second).

[0031] Step S200 further includes: S210: Collecting thermal imaging video data of the human eye state at a preset time interval to obtain a plurality of photos; S220: Mark the name of the human eye state for each photo, and all photos and their corresponding names form a frame data set; like Figure 2 As shown, in some embodiments, the preset convolutional neural network includes a first convolutional layer 101, a first batch normalization layer 102, a first pooling layer 103, a second convolutional layer 104, a second batch normalization layer 105, a second pooling layer 106, a third convolutional layer 107, a third batch normalization layer 108, a third pooling layer 109, a fourth convolutional layer 110, a fourth batch normalization layer 111, a fourth pooling layer 112, a first flattening layer 113, a first fully connected layer 114, a first regularization layer 115 and a second fully connected layer, which are connected in sequence.

[0032] The first convolution layer 101, the second convolution layer 104, the third convolution layer 107 and the fourth convolution layer 110 are all conventional two-dimensional convolution layers with a convolution kernel size of 3×3, and the first convolution layer 101 has 32 convolution kernels, the second convolution layer 104 has 64 convolution kernels, and the third convolution layer 107 and the fourth convolution layer 110 each have 128 convolution kernels; The first pooling layer 103, the second pooling layer 106 and the third pooling layer 109 are all two-dimensional maximum pooling layers with a size of 2×2. This structural design is intended to effectively extract the characteristics of the human eye state and optimize the learning ability of the model.

[0033] Step S300 further includes: S310: Randomly take 70% of the data set obtained in S200 as a training set and 30% as a test set.

[0034] S320: Use the processed data to train the pre-trained neural network model to obtain a trained neural network model.

[0035] In some embodiments, the provided folder contains two folders, each named after the state of the human eye, and each folder contains photos of the same state.

[0036] The role of the batch normalization layer is to standardize the data so that the subsequent network does not need to learn the data distribution of the previous layer, reduce the dependency between networks, and make the network converge faster. The principle is as follows: Calculate the mean of the mini-batch data:

[0037] Calculate the variance of the mini-batch data:

[0038] Standardize the minimum batch data to obtain new data with standard normal distribution:

[0039] The output updates to:

[0040] in, is a small positive number to prevent the denominator from being zero.

[0041] In some embodiments, the convolutional layer, the maximum pooling layer, and the batch normalization layer are respectively implemented using PyTorch In some embodiments, the convolutional layer, the maximum pooling layer, and the batch normalization layer are respectively implemented using the Conv2d, MaxPool2d, and BatchNormalization functions under the tf.keras module under the PyTorch framework.

[0042] In some embodiments, before using the processed data to train the convolutional neural network, it can be pre-trained using a public infrared human eye state data set to obtain the initial operating parameters of the neural network, and the initial operating parameters are assigned to the convolutional neural network to obtain a pre-trained neural network model. The pre-trained neural network model is then trained using the processed data to obtain a trained neural network model.

[0043] When training convolutional neural networks, the adaptive moment estimation optimization algorithm (Adma algorithm) can be used to find the optimal parameters and dynamically adjust the learning rate.

[0044] The process of the Adma algorithm is as follows: Get the new round of gradient values:

[0045] Update the first moment vector:

[0046] Update the second moment vector:

[0047] Compute the bias-corrected first moment vector:

[0048] Compute the bias-corrected second moment vector:

[0049] Update parameters: ,until convergence.

[0050] in: is the learning rate, initially set to 0.001; is the exponential decay rate of the moment estimate, and the initial value is , ; For parameters The random objective function value of ; (element-wise multiplication); small parameter To prevent the denominator from being 0, take , m t is the momentum estimate of the gradient; v t is the momentum estimate of the squared gradient; θ t is the parameter for the tth iteration.

[0051] The convolutional neural network of the present invention can effectively extract the basic features of the input image by using 32 3×3 convolution kernels in the first convolution layer, which increases the expressive power of the model. The second and third convolution layers use 64 and 128 3×3 convolution kernels respectively, further improving the depth and complexity of feature extraction; Each convolutional layer is followed by a batch normalization layer to reduce the dependencies between networks and promote faster convergence of the model, thereby improving training efficiency; The introduction of the maximum pooling layer effectively reduces the dimension of the feature map and reduces the computational complexity; The flattening layer flattens the multi-dimensional convolution output into one dimension to prepare data for subsequent fully connected layer processing; The first fully connected layer uses 128 neurons and is combined with a Dropout layer to prevent overfitting, thereby improving the generalization ability of the model; The output layer consists of 1 neuron, which uses the sigmoid activation function to output probability values, which is suitable for binary classification problems; In some embodiments, the network model is implemented by using the Keras API under the TensorFlow framework for efficient construction and training, and then the recognition function is implemented in the embedded system through the TensorFlow Lite Micro framework.

[0052] In an exemplary embodiment, the hardware configuration adopted by the present invention is: the central processing unit is AMD Ryzen 96900HX with Radeon Graphics eight-core, the memory size is 16GB, and the graphics card is an NVIDIA GeForce RTX3060 with a video memory of 8GB; the system and software configuration are: the system is Windows 11, the programming language is Python 3.8, and the network is built using the Tensorflow framework; the hyperparameters for training the 2D convolutional neural network are: the number of epochs for all sample training is 20, the number of training samples per batch is batchsize=32, and the initial learning rate is =0.001, and save the model every 20 epochs of training.

[0053] MCU deployment includes model conversion and uploading.

[0054] Step S400 further includes: S410: Set the converter to use integer quantization. The supported operation type is integer quantization. Both input and output are set to integer quantization.

[0055] S420: Convert the model and save the model to a file.

[0056] Make the model lightweight.

[0057] Step S500 implements the deployment algorithm to the MCU.

[0058] Step S500 further includes: S510: Use the TensorFlow Lite Micro framework to deploy the model to the project.

[0059] S520: Compile the code and upload it to the MCU.

[0060] Step S600 further includes: S610: The infrared camera is used to obtain the state of human eyes that needs to be monitored, and the video frames will be extracted. The extraction speed can be selected.

[0061] S620: Preprocess the acquired frame image, such as converting BGR to RGB, converting numpy array to PIL image, adjusting the image size, normalizing it and then sending it to the network model for recognition.

[0062] S630: Continuously refresh the frame data obtained from the video and display the predicted value of the video frame image to achieve real-time monitoring of the human eye state based on edge computing.

[0063] Specifically, the human eye is placed within the shooting range of the infrared camera, the embedded system acquires the video, extracts and continuously identifies the video frames, obtains the eye status and outputs it for display.

[0064] The infrared sensor human eye state monitoring method based on edge computing in the embodiment of the present invention obtains thermal imaging video data of human body movements through infrared sensors, can identify human body movements even in dark environments, and will not infringe on user privacy; the improved convolutional neural network and network structure are used to reduce the model size and realize operation on embedded systems.

[0065] like Figure 3 As shown, another embodiment of the present invention provides an infrared sensor human eye state monitoring system based on edge computing, including: The acquisition module 510 is configured to acquire thermal imaging video data of human eye status through an infrared sensor; A processing module 520 is configured to process the thermal imaging video data of the human eye state to obtain processed data; A training module 530 is configured to train a preset convolutional neural network using the processed data to obtain a trained neural network model; The lightweight module 540 converts the model into the TensorFlow Lite type using the full integer method; Deployment module 550, integrates the model into the project, loads and runs the model, compiles and uploads the recognition code; The recognition module 560 is configured to recognize the eye state of the human body to be recognized by the program running on the obtained MCU, and obtain a recognition result prediction result; The specific implementation methods of the acquisition module 510, the processing module 520, the training module 530, the lightweight module 540, the deployment module 550 and the identification module 560 are the same as those in the above embodiment and will not be repeated here.

[0066] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for monitoring human eye status using an infrared sensor based on edge computing, characterized in that: Includes steps: S100: Collects thermal imaging video data of human eye status through infrared sensors; S200: Processing the thermal imaging video data of human eyes to obtain an eye-opening and eye-closing picture data set; S300: Train the preset TinyCNN convolutional neural network using the processed photo data set to obtain a trained model; S400: Perform lightweight operations on the trained model based on the TensorFlow Lite framework to compress the model weights; S500: Use TensorFlow Lite Micro framework to deploy the model to MCU; S600: Cooperates with infrared sensors to realize real-time monitoring of human eye status based on edge computing, displays recognition prediction values ​​and updates them continuously, 0 means eyes closed, and 1 means eyes open.

2. The method for monitoring human eye status using an infrared sensor based on edge computing according to claim 1, characterized in that: The step S100 includes: S110: Arranging the infrared sensors according to the field of view of the infrared sensors so that the human eye area is located within the field of view of the infrared sensors; S120: enabling the infrared sensor to continuously collect frames of the human eye at a preset frame rate to obtain raw data of the human eye state, where the human eye state includes two categories: eyes open and eyes closed.

3. The method for monitoring human eye status using an infrared sensor based on edge computing according to claim 1, characterized in that: The step S200 includes: S210: Collecting thermal imaging video data of the human eye state at a preset time interval to obtain a plurality of photos; S220: Mark each photo with a name of the eye state, and all photos and their corresponding names form a frame data set.

4. The method for monitoring human eye status using an infrared sensor based on edge computing according to claim 1, characterized in that: The preset TinyCNN convolutional neural network includes a first convolutional layer, a first batch normalization layer, a first pooling layer, a second convolutional layer, a second batch normalization layer, a second pooling layer, a third convolutional layer, a third batch normalization layer, a third pooling layer, a fourth convolutional layer, a fourth batch normalization layer, a fourth pooling layer, a flattening layer, a first fully connected layer, a Dropout layer and an output layer, which are connected in sequence.

5. The method for monitoring human eye status using an infrared sensor based on edge computing according to claim 4 is characterized in that: The first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer are all conventional two-dimensional convolution layers with a convolution kernel size of 3×3, and the first convolution layer has 32 convolution kernels, the second convolution layer has 64 convolution kernels, and the third convolution layer and the fourth convolution layer have 128 convolution kernels; the first pooling layer, the second pooling layer and the third pooling layer are all two-dimensional maximum pooling layers with a size of 2×2.

6. The method for monitoring human eye status using an infrared sensor based on edge computing according to claim 1, characterized in that: The step S300 includes: S310: Randomly take 70% of the data set obtained in S200 as a training set and 30% as a test set; S320: Use the processed data to train the pre-trained neural network model, use the adaptive moment estimation optimization algorithm to find the optimal parameters, and obtain the trained neural network model.

7. The method for monitoring human eye status using an infrared sensor based on edge computing according to claim 1, characterized in that: The step S400 includes: S410: Set the converter to use integer quantization, the supported operation type is integer quantization, and both input and output are set to integer quantization; S420: Convert the model and save the model to a file.

8. The method for monitoring human eye status using an infrared sensor based on edge computing according to claim 1, characterized in that: The step S500 includes: S510: Use the TensorFlow Lite Micro framework to deploy the model into the project; S520: Compile the code and upload it to the MCU.

9. The method for monitoring human eye status using an infrared sensor based on edge computing according to claim 1, characterized in that: The step S600 includes: S610: obtaining the human eyes to be monitored through an infrared camera, extracting frames from the video, and selecting an extraction speed; S620: Preprocess the acquired frame image, convert the BGR format to RGB format, convert the numpy array to PIL image, adjust the image size, normalize it, and then send it to the network model for recognition; S630: Continuously refresh and display the frame data acquired from the video, and display the predicted value of the video frame image to achieve real-time monitoring of the human eye state based on edge computing.

10. An infrared sensor human eye status monitoring system based on edge computing, characterized in that: include: An acquisition module, used for acquiring thermal imaging video data of human eye status through an infrared sensor; A processing module, used for processing the thermal imaging video data of the human eye state to obtain processed data; A training module is used to train a preset convolutional neural network using the processed data to obtain a trained neural network model; Lightweight module, using the full integer method to convert the model to TensorFlow Lite type; Deploy the module, integrate the model into the project, load and run the model, compile and upload the recognition code; The recognition module is configured to recognize the eye state of the human body to be monitored by the program running on the obtained MCU to obtain the recognition prediction result.

Images