Non-perceived interaction method based on eye state recognition

Abstract

The invention discloses a non-perceived interaction method based on eye state recognition, and belongs to the field of man-machine interaction, and the method comprises the following steps: S1, selecting a proper neural network model; s2, creating a database; s3, training a neural network model; s4, converting the neural network model; the method has the advantages that a highly-optimized eye opening recognition algorithm is directly deployed to the local processor of the eye protection equipment, the design selection brings remarkable safety and efficiency advantages, dependence on network connection is abandoned, and the method is simple and convenient to operate. According to the technology, the eye state monitoring task can be accurately executed in real time in a completely offline state, and the fluency and continuity of user experience are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction, and more specifically to an imperceptible interaction method based on eye state recognition. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technology, the field of human-computer interaction has made significant progress. In the current technological environment, the human-computer interaction mode is undergoing a profound transformation from physical input to biometric recognition, and researchers are beginning to explore the use of human biological signals for contactless and imperceptible human-computer interaction.

[0003] Among them, the imperceptible interaction method based on eye status has received widespread attention as an emerging technical direction. This technology requires collecting eye status through devices and uploading it to the cloud through the network for algorithm reasoning. The cloud then transmits the signal to the device to form human-computer interaction.

[0004] However, there are still some challenges in the existing interactive technology based on eye status. For example, privacy protection is an aspect that needs to be considered in this field. The algorithm reasoning of traditional solutions is based on the cloud. The core is that the device must collect eye movement status and upload user images to the cloud, and then the cloud algorithm reasoning is implemented. This leads to the leakage of user images uploaded to the cloud. In addition, the interactive technology based on eye status is too dependent on the network. When the network is poor or there is no network, it seriously affects the use. Users can easily perceive the device status, resulting in extremely poor perception. Therefore, how to provide a natural and smooth interactive experience without infringing personal privacy is an important research direction based on eye status interactive technology.

[0005] In view of this, the inventors have designed an imperceptible interaction method based on eye state recognition. Summary of the invention

[0006] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a non-perceptual interaction method based on eye state recognition. The advantage of the present invention is that a highly optimized eye open recognition algorithm is directly deployed on the local processor of the eye protection device. This design choice brings significant security and efficiency advantages. By abandoning the dependence on network connection, the technology can perform eye state monitoring tasks in real time and accurately in a completely offline state, significantly improving the fluency and continuity of the user experience.

[0007] To achieve the above object, the present invention provides the following technical solution: a method for imperceptible interaction based on eye state recognition, comprising the following steps:

[0008] S1. Select a suitable neural network model: Select and establish a basic model, and adjust the model input and output according to the task requirements and MCU resources;

[0009] S2. Create a database: adjust the derived data set according to the camera parameters, shooting conditions, and multi-state color images;

[0010] S3. Training neural network model: classify the data set and train the model, and update and evaluate the model performance;

[0011] S4. Convert the neural network model: convert the storage format of the trained model and perform quantization, and then re-evaluate the model performance;

[0012] S5. Load model parameters into MCU: Load the model into MCU in file format.

[0013] The present invention is further preferred: the neural network model in step S1 includes the following steps:

[0014] S11.MobileNetV2: MobileNetV2 is selected as the base model because it is a lightweight neural network structure suitable for running on devices with limited resources;

[0015] S12. Adjust the model output tensor size according to task requirements: According to the actual application requirements, adjust the output dimension of the model to meet the expected results;

[0016] S13. Adjust model input tensors based on MCU resources: Considering the computing power and storage limitations of the MCU, it may be necessary to adjust the input size of the model.

[0017] The present invention is further preferred: the neural network model in step S2 includes the following steps:

[0018] S21. Determine specific camera parameters: Determine camera parameters for data collection in order to obtain high-quality training samples;

[0019] S22. Simulate the shooting angle during use: simulate various shooting angles in actual use scenarios to cover as many situations as possible;

[0020] S23. Multiple, multi-person, and multi-state image acquisition: Collect diverse image data, including different people, different lighting conditions, and different eye states;

[0021] S24. Derive datasets by adjusting light brightness, etc.: Generate additional data by changing lighting conditions to enhance the generalization ability of the model;

[0022] S25. Modify the image to match the input format supported by the model: Convert the collected image into the input format required by the model.

[0023] The present invention is further preferred: the neural network model in step S3 includes the following steps:

[0024] S31. Divide the dataset into a training set and a validation set: Divide the dataset into two parts, one for training the model and the other for evaluating the model performance;

[0025] S32. Sending the training set data to model training: using the training set data to train the neural network model;

[0026] S33. Evaluate model performance on the validation set: Use the validation set data to evaluate the accuracy and generalization ability of the model;

[0027] S34. Update evaluation model performance: Update model parameters based on evaluation results to improve performance.

[0028] The present invention is further preferred: the neural network model in step S4 includes the following steps:

[0029] S41. Storing the trained model in SaveModel format: saving the trained model in SaveModel format;

[0030] S42.SaveModel format to tflite format: Convert the model to TensorFlowLite format suitable for mobile devices;

[0031] S43. quantize the model and convert the parameters into 8-bit integer format: quantize and compress the model to reduce the size of the model;

[0032] S44. Re-evaluate model performance: Re-evaluate the performance of the model after quantization to ensure that the accuracy loss is acceptable.

[0033] The present invention is further preferred: the neural network model in step S5 includes the following steps:

[0034] S51. Export the neural network model parameters in .h file format: export the model parameters as .h file for easy use in C / C++ code;

[0035] S52.Introduce the .h file in the embedded project: Include the .h file in the embedded project to run the model on the MCU.

[0036] In summary, the present invention has the following advantages:

[0037] 1. By embedding the eye-opening recognition algorithm into the eye protection device, the user's eye biometric data does not need to be uploaded to the cloud or other external servers, which fundamentally reduces the security risks in the data transmission process and effectively prevents the possibility of data leakage or malicious attacks. In the current increasingly severe network security environment, it can build a solid line of defense for user data security, allowing users to enjoy convenient eye protection services without worrying about privacy leakage, thereby enhancing users' trust and acceptance of the product.

[0038] 2. By embedding the eye-opening recognition algorithm into the eye protection device, the device can be operated without networking. The algorithm can be directly operated on the device, avoiding network delays and ensuring the immediacy and accuracy of eye status recognition. In an environment with unstable network or no network coverage, the present invention can still work stably and provide users with uninterrupted personalized eye protection services. This high efficiency not only improves the user experience, but also expands the application scenarios of the technology, making it also play an important role in outdoor activities, remote areas or situations with limited network.

[0039] 3. Local processing reduces the need for external communication, which in turn reduces the energy consumption of the device and extends battery life. This is especially important for wearable devices and other products that have high requirements for battery life, allowing users to enjoy all-weather eye protection without frequent charging, which improves convenience and practicality.

[0040] Since it does not rely on network conditions, the eye protection device of the present invention is easier to promote to every corner of the world, including areas where network infrastructure is not yet perfect, which helps to narrow the digital divide and allow more people to benefit from advanced human-computer interaction technology and enjoy the improvement in the quality of healthy life brought by science and technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flowchart of neural network training and deployment of this embodiment;

[0042] Figure 2 It is a device interaction flow chart of this embodiment. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0044] An imperceptible interaction method based on eye state recognition, such as Figure 1 As shown, the following steps are included:

[0045] S1. Select a suitable neural network model: Select and establish a basic model, and adjust the model input and output according to the task requirements and MCU resources, including the following steps:

[0046] S11.MobileNetV2: MobileNetV2 is selected as the base model because it is a lightweight neural network structure suitable for running on devices with limited resources;

[0047] S12. Adjust the model output tensor size according to task requirements: According to the actual application requirements, adjust the output dimension of the model to meet the expected results;

[0048] S13. Adjust model input tensors based on MCU resources: Considering the computing power and storage limitations of the MCU, it may be necessary to adjust the input size of the model.

[0049] S2. Create a database: adjust the derived data set according to the camera parameters, shooting conditions, and multi-state color image conditions, including the following steps:

[0050] S21. Determine specific camera parameters: Determine camera parameters for data collection in order to obtain high-quality training samples;

[0051] S22. Simulate the shooting angle during use: simulate various shooting angles in actual use scenarios to cover as many situations as possible;

[0052] S23. Multiple, multi-person, and multi-state image acquisition: Collect diverse image data, including different people, different lighting conditions, and different eye states;

[0053] S24. Derive datasets by adjusting light brightness, etc.: Generate additional data by changing lighting conditions to enhance the generalization ability of the model;

[0054] S25. Modify the image to match the input format supported by the model: Convert the collected image into the input format required by the model.

[0055] S3. Training the neural network model: The data set is classified and the model is trained, and the model performance is updated and evaluated, including the following steps:

[0056] S31. Divide the dataset into a training set and a validation set: Divide the dataset into two parts, one for training the model and the other for evaluating the model performance;

[0057] S32. Sending the training set data to model training: using the training set data to train the neural network model;

[0058] S33. Evaluate model performance on the validation set: Use the validation set data to evaluate the accuracy and generalization ability of the model;

[0059] S34. Update evaluation model performance: Update model parameters based on evaluation results to improve performance.

[0060] S4. Convert the neural network model: Convert the trained model to a storage format and perform quantization, and then re-evaluate the model performance, including the following steps:

[0061] S41. Storing the trained model in SaveModel format: saving the trained model in SaveModel format;

[0062] S42.SaveModel format to tflite format: Convert the model to TensorFlowLite format suitable for mobile devices;

[0063] S43. quantize the model and convert the parameters into 8-bit integer format: quantize and compress the model to reduce the size of the model;

[0064] S44. Re-evaluate model performance: Re-evaluate the performance of the model after quantization to ensure that the accuracy loss is acceptable.

[0065] S5. Load model parameters into MCU: Load the model into MCU in file format, including the following steps:

[0066] S51. Export the neural network model parameters in .h file format: export the model parameters as .h file for easy use in C / C++ code;

[0067] S52.Introduce the .h file in the embedded project: Include the .h file in the embedded project to run the model on the MCU.

[0068] By using an MCU loaded with a neural network model, eye status recognition can be performed on the device without the need for a network, and interaction can be performed within the device.

[0069] like Figure 2 As shown, when used in eye massage equipment, most eye massage equipment requires users to close their eyes when using it, which can achieve better eye massage relaxation effect. The present invention can remind the user when the user opens his eyes. When the device detects the user's eye state, the signal is transmitted to the MCU inside the device, and the neural network model on the MCU makes a judgment. When the eyes are in an open state, a voice reminder is played; when the eyes are in a closed state, the eye detection continues.

[0070] Through this method, the eye massage device can detect the user's eye condition during operation, and play voice reminders when the user opens his eyes, helping the user to achieve better eye protection effects.

[0071] like Figure 2As shown, when this technology is applied to glasses or sunglasses, it can monitor the driver's eye state when driving a car. When the device detects the user's eye state, it transmits the signal to the MCU inside the device, and the neural network model on the MCU makes a judgment. When the eyes are in an open state, the eye detection continues; when the eyes are in a closed state, a voice reminder is played, so as to judge that the driver is in a fatigued state and give a timely reminder to avoid traffic accidents.

[0072] The working process and beneficial effects of the present invention are as follows:

[0073] Embedding the eye-opening recognition algorithm into the eye protection device means that the user's eye biometric data does not need to be uploaded to the cloud or other external servers, fundamentally reducing the security risks in the data transmission process and effectively preventing the possibility of data leakage or malicious attacks. In the current increasingly severe network security environment, it can build a solid line of defense for user data security, allowing users to enjoy convenient eye protection services without worrying about privacy leaks, thereby enhancing users' trust and acceptance of the product.

[0074] By embedding the eye-opening recognition algorithm into the eye protection device, the device can be operated without the need for an Internet connection. The algorithm can be run directly on the device, avoiding network delays and ensuring the immediacy and accuracy of eye status recognition. In an environment with unstable networks or no network coverage, the present invention can still work stably and provide users with uninterrupted personalized eye protection services. This high efficiency not only improves the user experience, but also expands the application scenarios of the technology, allowing it to play an important role in outdoor activities, remote areas, or situations where the network is restricted.

[0075] Local processing reduces the need for external communications, which in turn reduces the energy consumption of the device and extends battery life. This is especially important for products such as wearable devices that have high requirements for battery life, allowing users to enjoy all-weather eye protection without the need for frequent charging, improving convenience and practicality of use.

[0076] Since it does not rely on network conditions, the eye protection device of the present invention is easier to promote to every corner of the world, including areas where network infrastructure is not yet perfect, which helps to narrow the digital divide and allow more people to benefit from advanced human-computer interaction technology and enjoy the improvement in the quality of healthy life brought by science and technology.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the design concept of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for imperceptible interaction based on eye state recognition, characterized in that: The following steps are involved: S1. Select an appropriate neural network model: Choose to build a basic model and adjust the model input and output based on the task requirements and MCU resources; S2. Create a database: Adjust the derived dataset based on camera parameters, shooting conditions, and multi-state color images; S3. Training the neural network model: Classify the dataset and train the model, and update and evaluate the model performance; S4. Convert the neural network model: Convert the trained model to a different storage format, perform quantization, and then re-evaluate the model performance. S5. Load model parameters to MCU: Load the model into MCU in file format.

2. The method for imperceptible interaction based on eye state recognition according to claim 1, characterized in that: The neural network model in step S1 includes the following steps: S11.MobileNetV2: We chose MobileNetV2 as the base model because it is a lightweight neural network structure suitable for running on resource-constrained devices. S12. Adjust the model output tensor size according to task requirements: Based on actual application needs, adjust the model output dimension to meet the expected results; S13. Adjust model input tensors based on MCU resources: Considering the computing power and storage limitations of the MCU, it may be necessary to adjust the input size of the model.

3. The method for imperceptible interaction based on eye state recognition according to claim 1, characterized in that: The neural network model in step S2 includes the following steps: S21. Determine specific camera parameters: Determine camera parameters for data collection to obtain high-quality training samples; S22. Simulate shooting angles during use: Simulate various shooting angles in actual use scenarios to cover as many situations as possible; S23. Multiple, multi-person, and multi-state image acquisition: Collect diverse image data, including different people, different lighting conditions, and different eye states; S24. Dataset Derivation by Modulating Lighting, Etc.: Generate additional data by varying lighting conditions to enhance the generalization of the model. S25. Modify images to match the input format supported by the model: Convert the collected images into the input format required by the model.

4. The method for imperceptible interaction based on eye state recognition according to claim 1, characterized in that: The neural network model in step S3 includes the following steps: S31. Split the dataset into a training set and a validation set: Divide the dataset into two parts, one for training the model and the other for evaluating the model performance. S32. Sending the training set data to model training: using the training set data to train the neural network model; S33. Evaluate model performance on the validation set: Use the validation set data to evaluate the accuracy and generalization ability of the model; S34. Update evaluation model performance: Update model parameters based on evaluation results to improve performance.

5. The method for imperceptible interaction based on eye state recognition according to claim 1, characterized in that: The neural network model in step S4 includes the following steps: S41. Storing the trained model in SaveModel format: Save the trained model in SaveModel format; S42.SaveModel format to tflite format: Convert the model to TensorFlowLite format suitable for mobile devices; S43. Quantize the model and convert the parameters into 8-bit integer format: quantize and compress the model to reduce the model size; S44. Re-evaluate model performance: Re-evaluate the performance of the model after quantization to ensure that the accuracy loss is acceptable.

6. The method for imperceptible interaction based on eye state recognition according to claim 1, characterized in that: The neural network model in step S5 includes the following steps: S51. Export the neural network model parameters in .h file format: Export the model parameters to a .h file for easy use in C / C++ code; S52.Introduce the .h file into the embedded project: Include the .h file into the embedded project to run the model on the MCU.

Images