Eye opening recognition algorithm based on TinyML

Abstract

The invention discloses an eye opening recognition algorithm based on TinyML, and the algorithm comprises the following steps: S1, carrying out the matched hardware design, and enabling the hardware to comprise a USB charger, a battery, a DC-DC converter, an MCU microcontroller unit, a loudspeaker, an RTC clock, a key, an RF element, a camera, and an RGB indicating lamp; s2, training a TinyML model, and converting the model into a single-chip microcomputer support deployment form; model conversion, quantification and reasoning are carried out on the MCU microcontroller unit with extremely low power consumption and limited memory resources based on the TinyML model, customized design is carried out for a specific hardware architecture, the specific hardware design and a software algorithm are closely integrated, and the method is suitable for multiple fields such as intelligent wearable equipment and vehicle-mounted safety systems and has wide application prospects. And efficient and real-time monitoring and identification of the eye opening state can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of eyes-open recognition algorithms, and in particular to an eyes-open recognition algorithm based on TinyML. Background Art

[0002] In recent years, the leap of artificial intelligence technology has promoted the development of intelligent behavior monitoring technology, which plays a vital role in ensuring public safety and individual well-being. Existing eye-opening recognition technology mostly relies on complex image processing algorithms and high-performance processors. Common eye-opening recognition algorithm implementations can be roughly divided into two methods: one is to use the lower computer as an image acquisition device only, upload the image information to a higher-performance processor in the cloud through the network for inference, and then return the results to the lower computer; the other is to use hardware with higher computing power to perform inference offline, such as socs with GPUs;

[0003] The above two methods limit their application in resource-constrained environments, such as wearable devices and small embedded systems. To this end, we propose an eye-opening recognition algorithm based on TinyML. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes an open eyes recognition algorithm based on TinyML.

[0005] The present invention proposes an open eyes recognition algorithm based on TinyML, comprising the following steps:

[0006] S1: Design supporting hardware, including USB charger, battery, DC-DC converter, MCU microcontroller unit, speaker, RTC clock, button, RF radio frequency component, camera and RGB indicator light;

[0007] S2: Train the TinyML model and convert it into a form that can be deployed on a single chip. The specific steps include:

[0008] S201: Select a suitable neural network model: Select MobileNetV2 as the basis for training the neural network model, adjust the model output tensor according to the task requirements, and adjust the model input tensor according to the MCU microcontroller unit resources;

[0009] S202: Create a data set: determine the specific camera module and hardware parameters, install a simple test structure, simulate the relative position of the camera and the human eye when in use, collect images of the eye state multiple times and by multiple people, derive a data set from the collected images by adjusting the light brightness, and batch modify the collected images into the input tensor format supported by the model;

[0010] S203: training the neural network model: the data set in S202 is divided into a training set and a validation set at a ratio of 80% and 20%, the training set data is fed into the neural network model, the predicted value is calculated, the difference between the predicted value and the actual label is measured using the loss function, and the neural network model parameters are updated according to the gradient of the loss function. After repeating the execution for multiple cycles, the validation set data is fed into the neural network model to evaluate the performance of the neural network model, and the final validation result is required to be at a high accuracy level;

[0011] S204: Convert the neural network model: first store the neural network model trained in S203 in the SaveModel format, then convert it from the SaveModel format to the tflite format, and perform quantization operations on the neural network model to convert the parameters from high-precision floating point numbers to 8-bit integers to reduce the space occupied by the neural network model. After the conversion is completed, re-evaluate the accuracy of the neural network model through the validation set data to ensure that the accuracy loss is within an acceptable and reasonable range;

[0012] S205: Loading neural network model parameters into the MCU microcontroller unit: Exporting the neural network model parameters after quantization and conversion in S204 in .h format, introducing the .h file into the embedded project, and setting the array storing the model parameters to Flash storage;

[0013] S3: Configure the camera module in the MCU and perform model reasoning, which includes the following steps:

[0014] S301: Loading neural network model parameters, the specific steps are as follows:

[0015] S3011: Load model: load the pre-trained TFLite format model through the trained_tflite variable, and check the version number of the model to ensure that it is consistent with the version supported by TFLite Micro. If the version is inconsistent, terminate the program and output an error message;

[0016] S3012: Memory allocation: Allocate memory space from SPIRAM for tensors of TFLite format models. If the allocation fails, the program will output an error message and terminate. If the allocation succeeds, a prompt message indicating successful allocation will be output.

[0017] S3013: Register operation: Create a MicroMutableOpResolver instance, add the operation required by the TFLite format model to the MicroMutableOpResolver instance, and check whether each operation is successful. If it fails, terminate the program;

[0018] S3014: Create and configure the interpreter: Use the MicroInterpreter constructor to create an interpreter instance and allocate actual memory space for all tensors of the model by calling the AllocateTensors() method. If this step fails, the program will output an error message and terminate.

[0019] S3015: Get input and output tensor information: Save references to model input and output tensors to facilitate subsequent data input and result reading;

[0020] S302: Initialize the camera module so that it can transmit the video stream through the USB video class (UVC) interface. The specific steps are as follows:

[0021] S3021: Event group creation: Create an event group handle s_evt_handle for subsequent event management and synchronization. If the creation fails, the function will record the error information and trigger the assertion, interrupting the execution.

[0022] S3022: Memory allocation: allocate two double buffers, xfer_buffer_a and xfer_buffer_b, for USB data transmission, and allocate a frame buffer frame_buffer for storing JPEG format image frames. After each allocation, an assertion is used to ensure that the memory allocation is successful.

[0023] S3023: UVC configuration: construct the uvc_config_t structure, define the output parameters of the camera, and set the frame callback function camera_frame_cb to process each frame of data received;

[0024] S3024: Enable UVC function: Call uvc_streaming_config() function to pass the configured uvc_config to the underlying driver to initialize and enable UVC function. If the configuration fails, the function will record the error message and terminate the execution.

[0025] S303: Enable UVC streaming transmission, and enter the callback function for processing after receiving the image information. The specific steps are as follows:

[0026] S3031: First confirm that the received image frame format is MJPEG;

[0027] S3032: Use static variables jpeg_buffer and jpeg_new_buffer to save the decoded original image and the scaled image data, check whether there is enough free space in SPIRAM to store the two, and if there is enough free space, allocate the corresponding buffer on SPIRAM;

[0028] S3033: jpeg_buffer is allocated successfully, ESP-JPEG decoder is called to decode the MJPEG frame data, and the result is stored in jpeg_buffer;

[0029] S3034: Use the bilinear interpolation method resizeImageBilinear to scale the image size to the specified size, which conforms to the model input tensor form;

[0030] S3035: receiving the scaled and preprocessed image data as input, looping through each element of the input tensor, normalizing the input data, converting it from an 8-bit unsigned integer (range 0-255) to a floating point number (range 0-1), and further converting the normalized floating point number to a quantized int8_t data type, where the quantization scale is determined by quantization_scale;

[0031] S3036: Call interpreter->Invoke() method to execute the model reasoning process;

[0032] S3037: Obtain the quantized output of the model, perform dequantization on the inference result, and convert the quantized output back to a floating point format.

[0033] Preferably, in S201, the output tensor is adjusted to output only a floating point number in the range of 0-1, and the input tensor is adjusted to 27648, that is, an image with a resolution of 96*96 and a color format of RGB888.

[0034] Preferably, in S202, the hardware parameters of the camera include image frame format, color mode and focal length, and the resolution of the captured image is 320*240, and the image needs to be modified to a format with a resolution of 96*96 for input.

[0035] Preferably, in S3012, the memory space is called "tensor_arena" and is used to store intermediate results and tensor data of the model.

[0036] Preferably, in S3013, the MicroMutableOpResolver instance is used to register all computing operations required for the model, and its computing operations include but are not limited to: Softmax, Conv2D, Mul, Add, MaxPool2D, Reshape, FullyConnected and Logistic.

[0037] Preferably, in S3022, the size of the double buffer is at least equal to the size of a single frame image.

[0038] Preferably, in S3023, the output parameters of the camera include image width and height, frame rate, USB transmission buffer size and address, and frame buffer size and address.

[0039] Preferably, in S303, the callback processing function is responsible for receiving and processing the image frame data transmitted from the USB video class (UVC) interface, and its core tasks include decoding images in MJPEG format, image resizing preprocessing, and calling a machine learning model for reasoning analysis.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention performs model conversion, quantization and inference on the MCU microcontroller unit with extremely low power consumption and limited memory resources based on the TinyML model, and performs customized design for specific hardware architecture, tightly integrating the specific hardware design with the software algorithm. It is suitable for multiple fields such as smart wearable devices and vehicle safety systems, and can achieve efficient and real-time monitoring and recognition of the open eyes state. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a block diagram of the hardware connection in the TinyML-based eye-opening recognition algorithm proposed in the present invention;

[0043] Figure 2 A flowchart of training a TinyML model in an eye-opening recognition algorithm based on TinyML proposed by the present invention and converting the model into a form supported by a single-chip microcomputer for deployment;

[0044] Figure 3 This is a flowchart of configuring a camera module in a single-chip microcomputer and performing model reasoning in an eye-opening recognition algorithm based on TinyML proposed in the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further explained below in conjunction with specific embodiments.

[0046] Example

[0047] Reference Figure 1-3 , this embodiment proposes an open eyes recognition algorithm based on TinyML, including the following steps:

[0048] S1: Design supporting hardware, including USB charger, battery, DC-DC converter, MCU microcontroller unit, speaker, RTC clock, button, RF radio frequency component, camera and RGB indicator light;

[0049] The battery is connected to a USB charger and a DC-DC converter, and the DC-DC converter, the speaker, the RTC clock, the button, the RF radio frequency component, the camera and the RGB indicator light are all connected to the MCU microcontroller unit;

[0050] There are two buttons and two RGB indicators;

[0051] The DC-DC converter converts the voltage into 3.3V;

[0052] USB chargers are prohibited from being used while charging;

[0053] S2: Train the TinyML model and convert it into a form that can be deployed on a single chip. The specific steps include:

[0054] S201: Select a suitable neural network model: Select MobileNetV2 as the basis for training the neural network model, adjust the model output tensor according to the task requirements, and adjust the model input tensor according to the MCU microcontroller unit resources, where the output tensor is adjusted to output only a floating point number in the range of 0-1, and the input tensor is adjusted to 27648, that is, an image with a resolution of 96*96 and a color format of RGB888;

[0055] S202: Create a data set: determine the specific camera module and hardware parameters, install a simple test structure, simulate the relative position of the camera and the human eye when in use, collect images of the eye state multiple times and by multiple people, derive a data set from the collected images by adjusting the light brightness, and batch modify the collected images into the input tensor format supported by the model;

[0056] The hardware parameters of the camera include image frame format, color mode and focal length. The resolution of the captured image is 320*240, and the image needs to be modified to a resolution of 96*96 for input;

[0057] S203: training the neural network model: the data set in S202 is divided into a training set and a validation set at a ratio of 80% and 20%, the training set data is fed into the neural network model, the predicted value is calculated, the difference between the predicted value and the actual label is measured using the loss function, and the neural network model parameters are updated according to the gradient of the loss function. After repeating the execution for multiple cycles, the validation set data is fed into the neural network model to evaluate the performance of the neural network model, and the final validation result is required to be at a high accuracy level;

[0058] S204: Convert the neural network model: first store the neural network model trained in S203 in the SaveModel format, then convert it from the SaveModel format to the tflite format, and perform quantization operations on the neural network model to convert the parameters from high-precision floating point numbers to 8-bit integers to reduce the space occupied by the neural network model. After the conversion is completed, re-evaluate the accuracy of the neural network model through the validation set data to ensure that the accuracy loss is within an acceptable and reasonable range;

[0059] S205: Loading neural network model parameters into the MCU microcontroller unit: Exporting the neural network model parameters after quantization and conversion in S204 in .h format, introducing the .h file into the embedded project, and setting the array storing the model parameters to Flash storage;

[0060] S3: Configure the camera module in the MCU and perform model reasoning, which includes the following steps:

[0061] S301: Loading neural network model parameters, the specific steps are as follows:

[0062] S3011: Load model: load the pre-trained TFLite format model through the trained_tflite variable, and check the version number of the model to ensure that it is consistent with the version supported by TFLite Micro. If the version is inconsistent, terminate the program and output an error message;

[0063] S3012: Memory allocation: Allocate memory space from SPIRAM for tensors of TFLite format models. If the allocation fails, the program will output an error message and terminate. If the allocation succeeds, a prompt message indicating successful allocation will be output.

[0064] The memory space is called "tensor_arena", which is used to store the intermediate results and tensor data of the model;

[0065] S3013: Register operation: Create a MicroMutableOpResolver instance, add the operation required by the TFLite format model to the MicroMutableOpResolver instance, and check whether each operation is successful. If it fails, terminate the program;

[0066] The MicroMutableOpResolver instance is used to register all operations required by the model, including but not limited to: Softmax, Conv2D, Mul, Add, MaxPool2D, Reshape, FullyConnected, and Logistic;

[0067] S3014: Create and configure the interpreter: Use the MicroInterpreter constructor to create an interpreter instance and allocate actual memory space for all tensors of the model by calling the AllocateTensors() method. If this step fails, the program will output an error message and terminate.

[0068] S3015: Get input and output tensor information: Save references to model input and output tensors to facilitate subsequent data input and result reading;

[0069] S302: Initialize the camera module so that it can transmit the video stream through the USB video class (UVC) interface. The specific steps are as follows:

[0070] S3021: Event group creation: Create an event group handle s_evt_handle for subsequent event management and synchronization. If the creation fails, the function will record the error information and trigger the assertion, interrupting the execution.

[0071] S3022: Memory allocation: allocate two double buffers, xfer_buffer_a and xfer_buffer_b, the size of the double buffer is at least equal to the size of a single frame image, for USB data transmission, and allocate a frame buffer frame_buffer for storing JPEG format image frames. After each allocation, an assertion is used to ensure that the memory allocation is successful;

[0072] S3023: UVC configuration: construct the uvc_config_t structure, define the output parameters of the camera, and set the frame callback function camera_frame_cb to process each frame of data received. The output parameters of the camera include image width and height, frame rate, USB transmission buffer size and address, and frame buffer size and address;

[0073] S3024: Enable UVC function: Call uvc_streaming_config() function to pass the configured uvc_config to the underlying driver to initialize and enable UVC function. If the configuration fails, the function will record the error message and terminate the execution.

[0074] S303: Enable UVC streaming transmission, and enter the callback function for processing after receiving the image information. The callback processing function is responsible for receiving and processing the image frame data transmitted from the USB video class (UVC) interface. Its core tasks include decoding MJPEG format images, image size scaling preprocessing, and calling the machine learning model for reasoning analysis;

[0075] The specific steps are as follows:

[0076] S3031: First confirm that the received image frame format is MJPEG;

[0077] S3032: Use static variables jpeg_buffer and jpeg_new_buffer to save the decoded original image and the scaled image data, check whether there is enough free space in SPIRAM to store the two, and if there is enough free space, allocate the corresponding buffer on SPIRAM;

[0078] S3033: jpeg_buffer is allocated successfully, ESP-JPEG decoder is called to decode the MJPEG frame data, and the result is stored in jpeg_buffer;

[0079] S3034: Use the bilinear interpolation method resizeImageBilinear to scale the image size to the specified size, which conforms to the model input tensor form;

[0080] S3035: receiving the scaled and preprocessed image data as input, looping through each element of the input tensor, normalizing the input data, converting it from an 8-bit unsigned integer (range 0-255) to a floating point number (range 0-1), and further converting the normalized floating point number to a quantized int8_t data type, where the quantization scale is determined by quantization_scale;

[0081] S3036: Call interpreter->Invoke() method to execute the model reasoning process;

[0082] S3037: Obtain the quantized output of the model, perform dequantization processing on the inference result, and convert the quantized output back to a floating point format;

[0083] This embodiment performs model conversion, quantization, and inference on an MCU microcontroller unit with extremely low power consumption and limited memory resources based on the TinyML model, and performs customized design for a specific hardware architecture. It tightly integrates the specific hardware design with the software algorithm and is suitable for multiple fields such as smart wearable devices and vehicle safety systems, and can achieve efficient and real-time monitoring and recognition of open eyes.

[0084] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An open eyes recognition algorithm based on TinyML, characterized in that: The following steps are involved: S1: Design supporting hardware, including USB charger, battery, DC-DC converter, MCU microcontroller unit, speaker, RTC clock, button, RF radio frequency component, camera and RGB indicator light; S2: Train the TinyML model and convert it into a form that can be deployed on a single chip. The specific steps include: S201: Select a suitable neural network model: Select MobileNetV2 as the basis for training the neural network model, adjust the model output tensor according to the task requirements, and adjust the model input tensor according to the MCU microcontroller unit resources; S202: Create a data set: determine the specific camera module and hardware parameters, install a simple test structure, simulate the relative position of the camera and the human eye when in use, collect images of the eye state multiple times and by multiple people, derive a data set from the collected images by adjusting the light brightness, and batch modify the collected images into the input tensor format supported by the model; S203: training the neural network model: the data set in S202 is divided into a training set and a validation set at a ratio of 80% and 20%, the training set data is fed into the neural network model, the predicted value is calculated, the difference between the predicted value and the actual label is measured using the loss function, and the neural network model parameters are updated according to the gradient of the loss function. After repeating the execution for multiple cycles, the validation set data is fed into the neural network model to evaluate the performance of the neural network model, and the final validation result is required to be at a high accuracy level; S204: Convert the neural network model: first store the neural network model trained in S203 in the SaveModel format, then convert it from the SaveModel format to the tflite format, and perform quantization operations on the neural network model to convert the parameters from high-precision floating point numbers to 8-bit integers to reduce the space occupied by the neural network model. After the conversion is completed, re-evaluate the accuracy of the neural network model through the validation set data to ensure that the accuracy loss is within an acceptable and reasonable range; S205: Loading neural network model parameters into the MCU microcontroller unit: Exporting the neural network model parameters after quantization and conversion in S204 in .h format, introducing the .h file into the embedded project, and setting the array storing the model parameters to Flash storage; S3: Configure the camera module in the MCU and perform model reasoning, which includes the following steps: S301: Loading neural network model parameters, the specific steps are as follows: S3011: Load model: load the pre-trained TFLite format model through the trained_tflite variable, and check the version number of the model to ensure that it is consistent with the version supported by TFLite Micro. If the version is inconsistent, terminate the program and output an error message; S3012: Memory allocation: Allocate memory space from SPIRAM for tensors of TFLite format models. If the allocation fails, the program will output an error message and terminate. If the allocation succeeds, a prompt message indicating successful allocation will be output. S3013: Register operation: Create a MicroMutableOpResolver instance, add the operation required by the TFLite format model to the MicroMutableOpResolver instance, and check whether each operation is successful. If it fails, terminate the program; S3014: Create and configure the interpreter: Use the MicroInterpreter constructor to create an interpreter instance and allocate actual memory space for all tensors of the model by calling the AllocateTensors() method. If this step fails, the program will output an error message and terminate. S3015: Get input and output tensor information: Save references to model input and output tensors to facilitate subsequent data input and result reading; S302: Initialize the camera module so that it can transmit the video stream through the USB video class (UVC) interface. The specific steps are as follows: S3021: Event group creation: Create an event group handle s_evt_handle for subsequent event management and synchronization. If the creation fails, the function will record the error information and trigger the assertion, interrupting the execution. S3022: Memory allocation: allocate two double buffers, xfer_buffer_a and xfer_buffer_b, for USB data transmission, and allocate a frame buffer frame_buffer for storing JPEG format image frames. After each allocation, an assertion is used to ensure that the memory allocation is successful. S3023: UVC configuration: construct the uvc_config_t structure, define the output parameters of the camera, and set the frame callback function camera_frame_cb to process each frame of data received; S3024: Enable UVC function: Call uvc_streaming_config() function to pass the configured uvc_config to the underlying driver to initialize and enable UVC function. If the configuration fails, the function will record the error message and terminate the execution. S303: Enable UVC streaming transmission, and enter the callback function for processing after receiving the image information. The specific steps are as follows: S3031: First confirm that the received image frame format is MJPEG; S3032: Use static variables jpeg_buffer and jpeg_new_buffer to save the decoded original image and the scaled image data, check whether there is enough free space in SPIRAM to store the two, and if there is enough free space, allocate the corresponding buffer on SPIRAM; S3033: jpeg_buffer is allocated successfully, ESP-JPEG decoder is called to decode the MJPEG frame data, and the result is stored in jpeg_buffer; S3034: Use the bilinear interpolation method resizeImageBilinear to scale the image size to the specified size, which conforms to the model input tensor form; S3035: receiving the scaled and preprocessed image data as input, looping through each element of the input tensor, normalizing the input data, converting it from an 8-bit unsigned integer to a floating point number, and further converting the normalized floating point number to a quantized int8_t data type, where the quantization scale is determined by quantization_scale; S3036: Call interpreter->Invoke() method to execute the model reasoning process; S3037: Obtain the quantized output of the model, perform dequantization on the inference result, and convert the quantized output back to a floating point format.

2. The open eyes recognition algorithm based on TinyML according to claim 1, characterized in that: In S201, the output tensor is adjusted to output only a floating point number in the range of 0-1, and the input tensor is adjusted to 27648, that is, an image with a resolution of 96*96 and a color format of RGB888.

3. The open eyes recognition algorithm based on TinyML according to claim 1, characterized in that: In S202, the hardware parameters of the camera include image frame format, color mode and focal length. The resolution of the acquired image is 320*240, and the image needs to be modified to a format with a resolution of 96*96 for input.

4. The open eyes recognition algorithm based on TinyML according to claim 1, characterized in that: In the S3012, its memory space is called "tensor_arena", which is used to store the intermediate results and tensor data of the model.

5. The open eyes recognition algorithm based on TinyML according to claim 1, characterized in that: In S3013, the MicroMutableOpResolver instance is used to register all operations required for the model, and its operations include but are not limited to: Softmax, Conv2D, Mul, Add, MaxPool2D, Reshape, FullyConnected and Logistic.

6. The open eyes recognition algorithm based on TinyML according to claim 1, characterized in that: In S3022, the size of the double buffer is at least equal to the size of a single frame image.

7. The open eyes recognition algorithm based on TinyML according to claim 1, characterized in that: In the S3023, the output parameters of the camera include image width and height, frame rate, USB transmission buffer size and address, and frame buffer size and address.

8. The open eyes recognition algorithm based on TinyML according to claim 1, characterized in that: In S303, the callback processing function is responsible for receiving and processing the image frame data transmitted from the USB video class (UVC) interface, and its core tasks include decoding images in MJPEG format, image resizing preprocessing, and calling a machine learning model for reasoning analysis.

Images