Eye movement control system and method based on edge TPU chip model acceleration

Abstract

The invention provides an edge TPU chip model acceleration-based eye movement control system and method, and the system comprises a PC, a Docker, eye movement glasses and a carrying TPU chip, and the method comprises the steps: obtaining an outer scene image and an eye movement image through the eye movement glasses; a YOLOv5s model is modified and trained through a PC, the YOLOv5s model is converted into a Bmodel format through Docker format conversion and deployment so as to be compatible with a TPU chip, a code architecture is reconstructed based on an SAIL library and configured in the TPU chip, compatibility of the TPU chip and an eye movement controller is achieved, target position coordinates in an outer scene image are recognized through the Bmodel, and the target position coordinates in the outer scene image are recognized. A double-layer CNN convolution model is configured in the TPU chip to recognize the pupil center in the eye motion image, the pupil center is mapped to an outer scene to obtain a fixation point coordinate, and a control instruction is generated according to coordinate overlapping judgment. The problems of compatibility and integration of a TPU chip and an existing eye movement controller are solved, eyeball movement data of a user can be rapidly and accurately processed, meanwhile, the data processing precision is improved, and smoother and more natural user interaction experience is achieved.

Description

Technical Field

[0001] The present application relates to the field of smart glasses technology, and in particular to an eye movement control system and method based on edge TPU chip model acceleration. Background Art

[0002] Existing eye movement controllers have significant deficiencies in performance, energy efficiency, and user experience. First, these devices often face computing bottlenecks when processing high-frame-rate image data in real time, resulting in increased latency and decreased tracking accuracy. In addition, due to the high complexity and serious redundancy of the model, it is difficult to quickly complete object and pupil recognition. Secondly, the power consumption management of existing technologies is not optimized enough, and the battery life of portable devices is limited, which makes it difficult to meet the needs of long-term use. In addition, these devices have poor adaptability in complex environments (such as low light conditions or when users wear glasses), resulting in unstable tracking performance. Compatibility and integration issues also often limit the application of new technologies. Existing eye movement control systems are difficult to seamlessly integrate efficient new processors such as TPU chips, further limiting the improvement of device performance. Summary of the invention

[0003] In order to solve the technical problems that existing devices often face computing bottlenecks when processing high frame rate image data in real time, resulting in increased delays and decreased tracking accuracy, and it is difficult to quickly complete object and pupil recognition due to the high complexity and serious redundancy of the model, the present application provides an eye movement control system based on edge TPU chip model acceleration, including:

[0004] PC,is configured to perform model modification training of the YOLOv5s model;

[0005] Docker, configured to convert the YOLOv5s model to a Bmodel model and deploy it;

[0006] Eye-movement glasses are provided with a front camera and a pupil camera, the front camera is used to capture external scene images, and the pupil camera is used to capture eye movement images; the external scene images and the eye movement images are transmitted to the TPU chip via WIFI;

[0007] The TPU chip is internally configured with an eye tracking image processing module, a target recognition module, a coordinate judgment module and an instruction generation module. The eye tracking image processing module is configured to use a double-layer CNN convolution model to receive and recognize the pupil center in the eye movement image, and map the pupil center to the external scene image to obtain the gaze point coordinates; the target recognition module is configured to use the SAIL library to reconstruct the code structure, and use the Bmodel model to receive and recognize the target position coordinates in the external scene image; the coordinate judgment module is configured to perform coordinate overlap judgment on the gaze point coordinates and the target position coordinates; the instruction generation module is configured to generate control instructions based on the gaze point coordinates and the target position coordinates according to the result of the coordinate overlap judgment.

[0008] Through the above technical solutions, the compatibility and integration issues between TPU chips and existing eye movement controllers can be solved, which can significantly improve the performance and user experience of the eye movement control system. The TPU chip is designed to accelerate the calculation of machine learning models and can efficiently handle large-scale parallel computing tasks. By integrating the TPU chip with eye movement glasses, the eye movement control system can quickly and accurately process the user's eye movement data. Integration can not only improve the response speed of the system, but also improve the accuracy of data processing, thereby achieving a smoother and more natural user interaction experience. In addition, the efficient computing power of the TPU chip can also support more complex eye movement data analysis and pattern recognition, further improving the functionality and intelligence level of eye movement control. In Docker, the deep learning model is converted to the Bmodel model format supported by the TPU chip to provide faster response speed and higher efficiency, and the code architecture is reconstructed through the SAIL library to achieve computing acceleration, thereby realizing the compatibility solution between the TPU chip and the eye movement controller, and greatly reducing the power consumption and cost of the system. Traditional eye movement control systems often rely on CPU or GPU for data processing. Although they can meet basic needs, efficiency and energy consumption issues are prominent when processing complex algorithms. The efficient computing architecture of the TPU chip can complete the same computing tasks at lower power consumption, thereby extending the battery life of the device and reducing heat dissipation problems. At the same time, the modular design of the TPU chip makes it easy to integrate into existing systems without requiring major changes to the hardware architecture, reducing development and maintenance costs. By utilizing this feature of the TPU chip, the eye tracking controller can achieve more economical and efficient operation.

[0009] In some specific embodiments, the system includes a data buffer module and a data preprocessing module; the data buffer module is configured to temporarily store eye movement images and eye movement images captured by eye movement glasses, and transmit them to the data preprocessing module; the preprocessing module is configured to perform noise reduction, image enhancement and feature extraction operations on the eye movement images and eye movement images, and transmit the processed eye movement images and eye movement image data to the TPU chip.

[0010] Through the above technical scheme, the image data is preliminarily processed by the data buffer module and the data preprocessing module to reduce the burden of the TPU chip. The data buffer module acts as a temporary storage. Due to the inconsistency of the smooth data transmission rate, in order to prevent data loss or delay, the high frame rate image data obtained from the camera is temporarily stored to ensure that the data flows steadily into the TPU chip. The data preprocessing module performs preliminary processing on the data before entering the TPU chip, performs noise reduction, image enhancement, feature extraction and other operations, reduces the computational burden of the TPU chip, and enables it to focus on more complex deep learning tasks. This data preprocessing can not only improve the data quality and the accuracy of subsequent processing, but also reduce the workload and energy consumption of the TPU chip, thereby achieving more efficient and more accurate eye movement control. Through the collaborative work of the data buffer module and the data preprocessing module, the system can achieve fast, stable and efficient data processing, significantly improving the overall performance and user experience of the eye movement controller. In this way, the data can quickly enter the TPU chip for deep processing, ensuring the real-time and low latency of the entire processing flow. This not only improves the response speed of the system, but also ensures the accurate processing of high frame rate data, so that the eye movement controller can perform well in real-time applications.

[0011] In some specific embodiments, the front camera is an OV5640 camera, and the pupil camera is an 850nm infrared eye camera.

[0012] In some specific embodiments, a two-layer CNN convolutional model receives an eye movement image that has been resized to a grayscale image, generates 16 feature maps using a first layer of 5×5 convolution, outputs a feature map of size 60×60×16 through a ReLU activation function, reduces the 60×60×16 feature map to a size of 30×30×16 based on a 2×2 maximum pooling operation, generates 32 feature maps using a second layer of 5×5 convolution, outputs a feature map of size 30×30×32 through a ReLU activation function; converts the feature map of size 30×30×32 into a one-dimensional vector and inputs it into a fully connected layer, and outputs the gaze point coordinates using the fully connected layer.

[0013] Through the above technical solutions, the eye movement controller needs to maintain high performance and high precision in various complex environments, including different lighting conditions and users wearing glasses. To this end, it is crucial to optimize the structure of the two-layer CNN convolution model to improve the environmental adaptability of the system. By optimizing the structure of the two-layer CNN convolution model, the performance of the eye movement controller in complex environments can be significantly enhanced, ensuring that it can provide accurate and reliable eye movement control in various usage scenarios.

[0014] In some specific embodiments, the PC is configured to improve the C3 structure of the YOLOv5s model into a C2f structure, and add MobileNetv3 to the backbone network of the YOLOv5s model.

[0015] Through the above technical solutions, the YOLOv5s model is modified and trained on the PC to make the obtained model more suitable for the eye movement control system. This system modifies the training method of the model, optimizes the C3 structure of YOLOv5s, and adopts the C2f structure to enhance the richness of the gradient flow. The C2f structure draws on the design ideas of the C3 module and ELAN, so that the model can obtain richer gradient flow information while maintaining lightweight. After the model is modified, the recognition accuracy when the eye movement glasses are far away from the object can be improved. In addition, the YOLOv5s backbone network is modified, and MobileNetv3 is added. MobileNetv3 is used to achieve model lightweight and balance speed and accuracy. MobileNetv3 significantly reduces the computing cost and resource consumption while ensuring the balance between model speed and accuracy through its efficient convolution operation and module design. Through these optimization measures, the system can maintain efficient real-time performance when processing eye movement data, and provide high-accuracy recognition results in complex environments.

[0016] In some specific embodiments, Docker is configured to convert a YOLOv5s model whose original model format is PyTorch into a YOLOv5s model in int8_umodel format using a preset quantization standard tool, convert the YOLOv5s model in int8_umodel format into a Bmodel model using a preset script, load the Bmodel model into a TPU chip, and convert it into a set of instructions corresponding to the TPU chip.

[0017] Through the above technical solution, after converting to the Bmodel model in Dcoker, the TPU chip loads the Bmodel model, uses the relevant interface to send the input tensor to the TPU chip and retrieves the output tensor obtained after calculation, which can provide faster response speed and higher efficiency.

[0018] In some specific embodiments, the SAIL library is used to reconstruct the code structure, including allocating data processing tasks to the TPU chip in the pre-processing stage; performing quantization and pruning operations in the inference stage, and optimizing the processing of complex tasks using tensor operations; optimizing the result processing and storage process in the post-processing stage, and using parallel computing and memory optimization technology to process the results of the inference stage, wherein the post-processing stage includes result parsing, screening, and storage operations.

[0019] Through the above technical solutions, data processing tasks are assigned to the TPU chip in the pre-processing stage. By assigning these tasks to the TPU, the efficiency of data processing can be significantly improved, and the computing power of the TPU can be fully utilized. In the inference stage, the model is optimized by quantization and pruning, making the model lighter and more efficient while maintaining high precision. In addition, the tensor operation optimization function in the SAIL library enables complex computing tasks to be executed more efficiently on the TPU, thereby reducing the inference time. In the post-processing stage, the SAIL library is used to optimize the process of result processing and storage. The post-processing stage usually includes operations such as result parsing, screening and storage. The efficiency of these operations directly affects the response speed of the entire task. Through the parallel computing and memory optimization technology provided by the SAIL library, the inference results can be processed more quickly and stored efficiently in the specified location, greatly shortening the processing time.

[0020] In some specific embodiments, the system includes a power consumption management module, which is configured to monitor the real-time workload and performance requirements of the TPU chip. When it is detected that the TPU chip is in a state of processing a complex task, the operating frequency and voltage of the TPU chip are increased; when it is detected that the TPU chip is in a low-load state or an idle state, the operating frequency and voltage of the TPU chip are reduced; the system is also configured to collect TPU chip data when running tasks multiple times and perform statistical analysis on the data, so as to predict the task load and adjust the frequency and voltage of the TPU chip in advance.

[0021] Through the above technical solutions, power consumption management is an important technical issue when applying TPU chips to portable eye tracking controllers. Although TPU chips are more energy efficient than traditional processors in deep learning tasks, it is still crucial to further optimize power consumption to extend battery life in portable devices. Therefore, the operating frequency and power of the TPU chip are adjusted according to the real-time workload. The dynamic frequency adjustment technology can automatically adjust the operating frequency of the TPU according to the current processing task to reduce unnecessary power consumption. By evaluating the current load and dynamically adjusting the frequency and voltage of the TPU, it is ensured that the best balance is found between performance and energy efficiency. In addition, by collecting data from multiple running tasks and performing statistical analysis on them, it is possible to predict the task load and further optimize power consumption management by adjusting the frequency and voltage in advance. This method not only improves the computing performance of the TPU under high load, but also significantly reduces the power consumption under low load, effectively extending the battery life of the device and improving the overall efficiency of the portable eye tracking controller.

[0022] In some specific embodiments, the system includes a smart car, which is configured to receive the control instruction generated by the TPU chip and complete a preset action operation on the target position.

[0023] In some specific embodiments, the eye-tracking glasses are provided with an infrared filter.

[0024] Through the above technical solutions, adding infrared filters at the hardware level can provide more stable eye movement data under various lighting conditions.

[0025] According to the second aspect of the present invention, an eye movement control method based on edge TPU chip model acceleration implemented by the above system is proposed, comprising:

[0026] Use eye-tracking glasses to capture external scene images and eye movement images, and transmit them to the TPU chip via WIFI;

[0027] The double-layer CNN convolutional model of the TPU chip is used to receive and identify the pupil center in the eye movement image; the pupil center is mapped to the external scene image to obtain the gaze point coordinates; the Bmodel model of the TPU chip is used to receive and identify the target position coordinates in the external scene image; the gaze point coordinates and the target position coordinates are overlapped to determine the coordinates; and according to the result of the coordinate overlap determination, control instructions are generated based on the gaze point coordinates and the target position coordinates.

[0028] The present invention proposes an eye movement control system and method based on edge TPU chip model acceleration, and the technical effects thereof are:

[0029] The external scene image and eye movement image are obtained through the eye-tracking glasses; the YOLOv5s model is modified and trained through the PC, and the YOLOv5s model is converted into the Bmodel model through Docker. The code architecture is reconstructed based on the SAIL library and configured in the TPU chip, which realizes computing acceleration and greatly reduces the power consumption and cost of the system, thereby achieving compatibility between the TPU chip and the eye-tracking controller. The target position coordinates in the external scene image are identified through the YOLOv5s model, and the pupil center in the eye movement image is configured in the TPU chip. It is mapped to the external scene to obtain the gaze point coordinates, and the control instructions are generated according to the coordinate overlap judgment. Solving the compatibility and integration problems of the TPU chip and the existing eye-tracking controller can quickly and accurately process the user's eye movement data, while also improving the accuracy of data processing, and achieving a smoother and more natural user interaction experience. The TPU chip is designed to accelerate the calculation of machine learning models and can efficiently handle large-scale parallel computing tasks. By integrating the TPU chip with the eye-tracking glasses, the eye-tracking control system can quickly and accurately process the user's eye movement data. Integration can not only improve the response speed of the system, but also improve the accuracy of data processing, thereby achieving a smoother and more natural user interaction experience. In addition, the efficient computing power of the TPU chip can also support more complex eye movement data analysis and pattern recognition, further improving the functionality and intelligence of eye movement control. Traditional eye movement control systems often rely on CPU or GPU for data processing. Although they can meet basic needs, efficiency and energy consumption issues are prominent when processing complex algorithms. The efficient computing architecture of the TPU chip can complete the same computing tasks at lower power consumption, thereby extending the battery life of the device and reducing heat dissipation problems. At the same time, the modular design of the TPU chip makes it easy to integrate into existing systems without major changes to the hardware architecture, reducing development and maintenance costs. By leveraging this feature of the TPU chip, the eye movement controller can achieve more economical and efficient operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present application. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0031] Figure 1 is a schematic diagram of an eye movement control system based on edge TPU chip model acceleration according to an embodiment of the present application;

[0032] Figure 2This is a schematic diagram of the C2f structure of an embodiment of the present application;

[0033] Figure 3 This is a schematic diagram of the YOLOv5s model structure of an embodiment of the present application;

[0034] Figure 4 It is a schematic diagram of the operation of the system modules of an embodiment of the present application;

[0035] Figure 5 This is a flow chart of the operation of an eye movement control method based on edge TPU chip model acceleration according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0037] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] In response to the shortcomings of the existing technology, this application aims to solve several key problems: first, by optimizing the deep learning algorithm, improve the real-time and accuracy of eye movement control; second, improve the data transmission and processing links to achieve efficient real-time data processing and reduce system latency; third, through dynamic power management strategies, optimize the energy consumption of portable devices and extend battery life. In addition, this application will also improve the adaptability of the system under different environmental conditions, including changes in lighting and when the user wears glasses, to ensure the stability and accuracy of eye movement control. Finally, solve the compatibility and integration issues of TPU with existing systems to ensure that the new technology can be successfully applied to the eye movement controller.

[0039] Figure 1 : is a schematic diagram of an eye movement control system based on edge TPU chip model acceleration according to an embodiment of the present application, the system comprising:

[0040] PC100, configured for model modification training of the YOLOv5s model;

[0041] Docker200, configured to convert the YOLOv5s model to the Bmodel model and deploy it;

[0042] The eye-movement glasses 300 are provided with a front camera 301 and a pupil camera 302, wherein the front camera 301 is used to capture external scene images, and the pupil camera 302 is used to capture eye movement images; the external scene images and the eye movement images are transmitted to the TPU chip via WIFI;

[0043] In this embodiment, the front camera adopts the OV5640 camera, and the pupil camera adopts the 850nm infrared eye camera.

[0044] The TPU chip 400 is internally configured with an eye tracking image processing module 401, a target recognition module 402, a coordinate judgment module 403 and an instruction generation module 404. The eye tracking image processing module 401 is configured to use a double-layer CNN convolution model to receive and recognize the pupil center in the eye movement image, and map the pupil center to the external scene image to obtain the gaze point coordinates; the target recognition module 402 is configured to use the SAIL library to reconstruct the code structure, and use the Bmodel model to receive and recognize the target position coordinates in the external scene image; the coordinate judgment module 403 is configured to perform coordinate overlap judgment on the gaze point coordinates and the target position coordinates; the instruction generation module 404 is configured to generate control instructions based on the gaze point coordinates and the target position coordinates according to the result of the coordinate overlap judgment.

[0045] The smart car 500 is configured to receive control instructions generated by the TPU chip 400 and complete preset action operations on the target position.

[0046] The front camera 301 of the eye-tracking glasses 300 captures the user's eye movement image data, and the pupil camera 302 captures the eye movement image. The TPU chip 400 is equipped with an eye-tracking image processing module 401 including a double-layer CNN convolution model to process the image data, so as to realize accurate recognition and tracking of the pupil center, and map the pupil center to the external scene, so as to realize the accurate correspondence between the gaze point coordinates and the external scene. The Bmodel model converted by the PyTorch model in the target recognition module 402 is combined to perform real-time object recognition processing. The system can identify the target object in the real-time video, obtain the target position coordinates, compare and judge the target position coordinates with the gaze point coordinates, and then realize the interaction between the user and the environment, and generate control instructions from the eye movement control and target recognition results and transmit them to the smart car 500, so that it can perform predetermined action operations according to the target object selected by the user's eye movement. The model structure is modified by PC100, and Docker200 is converted to the Bmodel model supported by TPU chip 400 and deployed to provide faster response speed and higher efficiency; the code structure is reconstructed by using SAIL library in TPU chip 400 to achieve computing acceleration, so as to solve the compatibility and integration problems between TPU chip 400 and existing eye movement controllers, which can significantly improve the performance and user experience of eye movement control system. TPU chip 400 is designed to accelerate the calculation of machine learning models and can efficiently handle large-scale parallel computing tasks. By integrating TPU chip 400 with eye movement glasses 300, the system can quickly and accurately process the user's eye movement data. At the same time, the integration can not only improve the response speed of the system, but also improve the accuracy of data processing, so as to achieve a smoother and more natural user interaction experience. In addition, the efficient computing power of TPU chip 400 can also support more complex eye movement data analysis and pattern recognition, further improving the functionality and intelligence level of eye movement control. Traditional eye movement control systems often rely on CPU or GPU for data processing. Although they can meet basic needs, efficiency and energy consumption issues are prominent when processing complex algorithms. The efficient computing architecture of the TPU chip 400 can complete the same computing tasks at lower power consumption, thereby extending the battery life of the device and reducing heat dissipation problems. At the same time, the modular design of the TPU chip 400 makes it easy to integrate into existing systems without requiring major changes to the hardware architecture, thereby reducing development and maintenance costs. By utilizing this feature of the TPU chip 400, the eye movement control system can achieve more economical and efficient operation.

[0047] Specifically, this system uses the YOLOv5s deep learning model through PC100 training and modifies the deep learning model to make the obtained model more suitable for the eye movement control system. The C3 structure of the YOLOv5s model is improved to the C2f structure, and the C2f structure is used to enhance the richness of the gradient flow. The C2f structure draws on the design ideas of the C3 module and ELAN. Its structure is as follows Figure 2 As shown in the figure, the model can obtain richer gradient flow information while maintaining its lightweight. After the model is modified, the recognition accuracy can be improved when the eye-tracking glasses 300 are far away from the object. In addition, the YOLOv5 backbone network is modified, and MobileNetv3 is added. MobileNetv3 is used to achieve model lightweight and balance speed and accuracy. MobileNetv3 significantly reduces computing costs and resource consumption while ensuring a balance between model speed and accuracy through its efficient convolution operations and module design. The modified YOLOv5s model structure is shown in the figure below. Figure 3 As shown, through these optimization measures, the system can maintain efficient real-time performance when processing eye movement data, and provide highly accurate recognition results in complex environments.

[0048] In addition, this application uses the Shaolin development board equipped with the third-generation TPU chip 400BM1684 independently developed by Suanneng. The INT8 computing power of this TPU is as high as 17.6TOPS, and it supports multi-channel full HD video hardware encoding and decoding. It is compatible with various interfaces or peripherals such as cameras, screens, WiFi, and Ethernet, so it can be applied to the model acceleration of the eye movement control system. This system is realized through this development board. The system module operation diagram is shown in the figure below. Figure 4 As shown, the YOLOv5s model in PyTorch format is converted into a Bmodel model, configured in the TPU chip 400, interacts with the user and the external environment through the eye-tracking glasses 300 (AR glasses), and is sent to the Shaolin development board. The data calculation results are obtained through the interaction between the Shaolin development board and the TPU chip 400, and the control instructions are generated and sent to the smart car 500. The TPU chip 400 is applied to the eye-tracking controller. First, the deep learning algorithm is optimized to fully utilize the parallel computing power and matrix operation performance of the TPU chip 400. The first step of the optimization algorithm is to reconstruct the model structure, adjust the network level, and remove unnecessary parameters and levels to improve the computing efficiency. By streamlining the number of convolutional layers and optimizing the configuration of the pooling layer, it is ensured that the model reduces the amount of calculation while retaining accuracy. In addition, data parallel processing is the key to improving efficiency. By processing data in batches, the parallel computing power of the TPU chip 400 can be fully utilized. These measures can not only improve the real-time and accuracy of eye-tracking control, but also maximize the hardware advantages of the TPU chip 400 and improve the operating efficiency of the entire system.

[0049] Specifically, the eye movement control system needs to process the high frame rate image data from the eye movement glasses 300 in real time to ensure accurate eye movement control. This requires the construction of an efficient data transmission and processing link to ensure that the TPU chip 400 can quickly process this data. In order to further improve efficiency, data buffering and preprocessing modules are set before data transmission. These modules are responsible for preliminary processing of image data and reducing the burden on the TPU chip 400. Due to the inconsistency of the smooth data transmission rate, in order to prevent data loss or delay, the high frame rate image data obtained from the eye movement glasses 300 is temporarily stored, and the data buffer module is configured to temporarily store the eye movement images and eye movement images captured by the eye movement glasses 300, and transmit them to the data preprocessing module to ensure that the data flows steadily into the TPU chip 400; the preprocessing module is configured to perform noise reduction, image enhancement and feature extraction operations on the eye movement images and eye movement images, and transmit the processed eye movement images and eye movement images to the TPU chip 400, reducing the computational burden of the TPU chip 400, so that it can focus on more complex deep learning tasks. This data preprocessing can not only improve the data quality and the accuracy of subsequent processing, but also reduce the workload and energy consumption of the TPU chip 400, thereby achieving more efficient and precise eye movement control. Through the collaborative work of the data buffering and preprocessing modules, the system can achieve fast, stable and efficient data processing, significantly improving the overall performance and user experience of the eye movement controller. In this way, the data can quickly enter the TPU chip 400 for deep processing, ensuring the real-time and low latency of the entire processing flow. This not only improves the response speed of the system, but also ensures the accurate processing of high frame rate data, enabling the eye movement controller to perform well in real-time applications.

[0050] Specifically, when the TPU chip 400 is applied to a portable eye tracking controller, power consumption management is an important technical issue. Although the energy efficiency of the TPU chip 400 is higher than that of traditional processors in deep learning tasks, it is still crucial to further optimize power consumption in portable devices to extend battery life. Therefore, a power consumption management module based on a load-aware dynamic adjustment method is adopted. The power consumption management module is configured to monitor the real-time workload and performance requirements of the TPU chip 400. When it is detected that the TPU chip 400 is in a state of processing complex tasks, the operating frequency and voltage of the TPU chip 400 are increased to provide higher computing power and speed to ensure that the task is completed smoothly; when it is detected that the TPU chip 400 is in a low-load state or an idle state, the operating frequency and voltage of the TPU chip 400 are reduced to reduce unnecessary energy consumption. By evaluating the current load and dynamically adjusting the frequency and voltage of the TPU chip 400, it is ensured that the best balance is found between performance and energy efficiency; it is also configured to collect TPU chip 400 data when running tasks multiple times and perform statistical analysis on it, so as to predict the task load, adjust the frequency and voltage of the TPU chip 400 in advance, and further optimize power consumption management. This method not only improves the computing performance of the TPU chip 400 under high load, but also significantly reduces the power consumption under low load, effectively extending the battery life of the device and improving the overall efficiency of the portable eye tracking controller.

[0051] Specifically, in order to seamlessly integrate the TPU chip 400 into the eye movement controller, it is necessary to solve the compatibility issues with existing hardware and software systems. This involves designing compatibility and integration frameworks to ensure that the TPU chip 400 can work efficiently with other system components. The first step is to convert the model into a model format supported by the TPU chip 400. In Docker200, the PyTorch model is loaded into the script provided by Suanfeng, and the format is converted and processed. It is converted to int8_umodel through the quantization calibration tool. To reduce the model size and accelerate the inference speed, the weights and activation values ​​are quantized into a lower precision representation. Use the tools and scripts provided by Suanfeng to convert int8_umodel to the final Bmodel model. The Bmodel model is loaded onto the TPU chip 400 and converted into a set of TPU chip 400 instructions, so that the inference task can be performed on the TPU chip 400. The Bmodel model is loaded onto the TPU chip 400, and the input data is sent through the running interface to obtain the inference results. This process is a completely offline process. After loading the Bmodel model, the input tensor is sent to the TPU chip 400 using the relevant interface and the output tensor obtained after calculation is taken out. This method provides faster response speed and higher efficiency. By reconstructing the code architecture, the efficiency of image data pre-processing, model inference and result post-processing is significantly improved. The introduction of the SAIL library not only improves the performance of the overall code, but also achieves computational acceleration in multiple key links. In the pre-processing stage, the data loading and pre-processing process is optimized through the SAIL library. Traditional methods usually rely on the CPU for data processing, while the SAIL library allows the TPU chip 400 (Tensor Processing Unit) to be efficiently used to perform these tasks. This not only reduces the burden on the CPU, but also significantly improves the data processing speed. By allocating these tasks to the TPU chip 400, the efficiency of data processing can be significantly improved, and the computing power of the TPU chip 400 is fully utilized. In the inference stage, the model is optimized by quantization and pruning, making the model lighter and more efficient while maintaining high accuracy. In addition, by utilizing the tensor operation optimization function in the SAIL library, complex computing tasks can be executed more efficiently on the TPU chip 400, thereby reducing the inference time. In the post-processing stage, the SAIL library is used to optimize the process of result processing and storage. The post-processing stage usually includes operations such as result parsing, screening, and storage. The efficiency of these operations directly affects the response speed of the entire task. Through the parallel computing and memory optimization technology provided by the SAIL library, the inference results can be processed more quickly and stored efficiently in the specified location, greatly shortening the processing time.

[0052] Specifically, the eye movement controller needs to maintain high performance and high precision in various complex environments, including different lighting conditions and users wearing glasses. To this end, it is crucial to optimize the algorithm and hardware design to improve the environmental adaptability of the system. The specific technical solution includes training and optimizing a special deep learning model so that it can accurately identify eye movements under low light conditions or when the user wears glasses. This requires the use of a large amount of training data in different environments to enhance the robustness of the model. In addition, adding an infrared filter at the hardware level can provide more stable eye movement data under various lighting conditions. These optimization measures can significantly enhance the performance of the eye movement controller in complex environments and ensure that it can provide accurate and reliable eye movement control in various usage scenarios. The present application uses the eye camera of the eye movement glasses 300 to capture pupil images, and the module program will collect pupil data in real time after it is turned on, and the program has built-in judgment to prevent the acquisition of erroneous images. The front camera 301 is used to identify the target in the environment and when the object the pupil is looking at is consistent with the object recognized by the front camera 301, the capture action is then implemented. The double-layer CNN convolution model is used to input the image. After training with the collected data set, the model can calculate the pixel coordinates of the front camera of the gaze point based on the input eye image. The structure and data flow of the model are as follows: First, the input eye image is 640×480. After resizing, it is reduced to a grayscale image of 64×64×1, and then sent to the neural network model for calculation. The first layer of convolution uses a 5×5 convolution kernel to generate 16 feature maps. After the convolution operation, the ReLU activation function is used to increase the nonlinear expression ability of the model. The size of the output feature map is 60×60×16. Then it is pooled again. The 2×2 maximum pooling operation is used to further reduce the size of the feature map, reduce the amount of calculation, and extract more important features. The output size after pooling is 30×30×16. Then it enters the second convolution layer. The second convolution layer uses a 5×5 convolution kernel to generate 32 feature maps. The ReLU activation function is used again. The size of the output feature map is 30×30×32. Finally, after passing through the fully connected layer, the fully connected layer flattens the pooled feature map and converts it into a one-dimensional vector. Then it is input into the fully connected layer for further feature combination and calculation. The output of the fully connected layer is the pixel coordinates (x, y) of the target object in the front camera 301, which is the final prediction result of the model.

[0053] Specifically, after the eye movement interaction system of the present application completes the target object recognition and eye movement selection in the environment, it is necessary to transmit the obtained results to the smart car 500 to complete the operation of the target object. This system uses the python Socket library to complete the communication between the Shaolin development board and the smart car 500. Socket is an abstract layer between the application layer and the transport layer. It provides an interface for accessing the TCP / IP protocol family and is usually used to implement communication between the client and the server in network programming. In the TCP / IP protocol, Socket is identified by IP address, port number and protocol type. The IP address of the Raspberry Pi smart car 500 is used as the unique host identifier, and the port number indicates the specific network application on the host. The protocol type is TCP protocol.

[0054] According to a second aspect of the present invention, an eye movement control method implemented by the above system is proposed, such as Figure 5 As shown, including:

[0055] Preload the model;

[0056] Using eye-tracking glasses 300 to capture external scene images and eye movement images, and transmitting them to the TPU chip via WIFI;

[0057] The double-layer CNN convolution model of the TPU chip 400 is used to receive and identify the pupil center in the eye movement image; the pupil center is mapped to the external scene image to obtain the gaze point coordinates; the Bmodel model of the TPU chip 400 is used to receive and identify the target position coordinates in the external scene image; the gaze point coordinates and the target position coordinates are overlapped to determine the coordinates; according to the result of the coordinate overlap determination, a control instruction (action instruction) is generated based on the gaze point coordinates and the target position coordinates;

[0058] The smart car 500 performs actions according to the control instructions.

[0059] The system of this application achieves the following technical effects:

[0060] The external scene image and eye movement image are obtained through the eye-tracking glasses 300; the YOLOv5s model is modified and trained through PC100 to be compatible with the TPU chip 400, and the YOLOv5s model is converted to the Bmodel model format through Docker200 format conversion and deployment. The code architecture is reconstructed based on the SAIL library and configured in the TPU chip 400, which realizes computing acceleration and greatly reduces the power consumption and cost of the system, thereby achieving compatibility between the TPU chip 400 and the eye-tracking controller, and the target position coordinates in the external scene image are identified through the YOLOv5s model. A double-layer CNN convolution model is configured in the TPU chip 400 to identify the pupil center in the eye movement image, and mapped to the external scene to obtain the gaze point coordinates, and control instructions are generated according to the coordinate overlap judgment. Solving the compatibility and integration problems between the TPU chip 400 and the existing eye-tracking controller can quickly and accurately process the user's eye movement data, while also improving the accuracy of data processing, and achieving a smoother and more natural user interaction experience. The TPU chip 400 is designed to accelerate the calculation of machine learning models and can efficiently handle large-scale parallel computing tasks. By integrating the TPU chip 400 with the eye-tracking glasses 300, the eye-tracking control system can process the user's eye movement data quickly and accurately. Integration can not only improve the response speed of the system, but also improve the accuracy of data processing, thereby achieving a smoother and more natural user interaction experience. In addition, the efficient computing power of the TPU chip 400 can also support more complex eye-tracking data analysis and pattern recognition, further improving the functionality and intelligence level of eye-tracking control. Traditional eye-tracking control systems often rely on CPUs or GPUs for data processing. Although they can meet basic needs, efficiency and energy consumption issues are prominent when processing complex algorithms. The efficient computing architecture of the TPU chip 400 can complete the same computing tasks at lower power consumption, thereby extending the battery life of the device and reducing heat dissipation problems. At the same time, the modular design of the TPU chip 400 makes it easy to integrate into existing systems without requiring major changes to the hardware architecture, reducing development and maintenance costs. By utilizing this feature of the TPU chip 400, the eye-tracking controller can achieve more economical and efficient operation.

[0061] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0062] The modules involved in the embodiments described in the present application can be implemented by software or hardware. The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in this application (but not limited to) to form a technical solution.

Claims

1. An eye movement control system based on edge TPU chip model acceleration, characterized in that: include: PC,is configured to perform model modification training of the YOLOv5s model; Docker, configured to convert the YOLOv5s model into a Bmodel model and deploy it; Eye-movement glasses, provided with a front camera and a pupil camera, wherein the front camera is used to capture external scene images, and the pupil camera is used to capture eye movement images; Transmitting the external scene image and the eye movement image to the TPU chip via WIFI; The TPU chip is internally configured with an eye tracking image processing module, a target recognition module, a coordinate judgment module and an instruction generation module. The eye tracking image processing module is configured to receive and recognize the pupil center in the eye movement image using a double-layer CNN convolution model, and map the pupil center to an external scene image to obtain the gaze point coordinates; the target recognition module is configured to reconstruct the code structure using the SAIL library, and receive and recognize the target position coordinates in the external scene image using the Bmodel model; the coordinate judgment module is configured to perform coordinate overlap judgment between the gaze point coordinates and the target position coordinates; The instruction generation module is configured to generate a control instruction based on the gaze point coordinates and the target position coordinates according to the result of the coordinate overlap judgment.

2. The system according to claim 1, characterized in that The system comprises a data buffer module and a data preprocessing module; the data buffer module is configured to temporarily store the eye movement image and the eye movement image captured by the eye movement glasses, and transmit them to the data preprocessing module; The preprocessing module is configured to perform noise reduction, image enhancement and feature extraction operations on the eye movement image and the eye movement image, and transmit the processed eye movement image and the eye movement image data to the TPU chip.

3. The system according to claim 1, characterized in that The front camera is an OV5640 camera, and the pupil camera is an 850nm infrared eye camera.

4. The system according to claim 1, characterized in that The double-layer CNN convolution model receives the eye movement image after being resized to a grayscale image, generates 16 feature maps using the first layer 5×5 convolution, outputs a feature map with a size of 60×60×16 through the ReLU activation function, reduces the 60×60×16 feature map to a size of 30×30×16 based on a 2×2 maximum pooling operation, generates 32 feature maps using the second layer 5×5 convolution, outputs a feature map with a size of 30×30×32 through the ReLU activation function; converts the feature map with a size of 30×30×32 into a one-dimensional vector and inputs it into a fully connected layer, and outputs the gaze point coordinates using the fully connected layer.

5. The system according to claim 1, characterized in that The PC is configured to improve the C3 structure of the YOLOv5s model into a C2f structure, and to add MobileNetv3 to the backbone network of the YOLOv5s model.

6. The system according to claim 1, characterized in that The Docker is configured to convert the YOLOv5s model whose original model format is PyTorch into a YOLOv5s model in int8_umodel format using a preset quantization standard tool, convert the YOLOv5s model in int8_umodel format into a Bmodel model using a preset script, load the Bmodel model into the TPU chip, and convert it into a set of corresponding instructions for the TPU chip.

7. The system according to claim 1, characterized in that The code structure is reconstructed by using the SAIL library, including allocating data processing tasks to the TPU chip in the pre-processing stage; performing quantization and pruning operations in the inference stage, and optimizing the processing of complex tasks by using tensor operations; In the post-processing stage, the process of result processing and storage is optimized, and the results of the reasoning stage are processed using parallel computing and memory optimization technology, wherein the post-processing stage includes result parsing, screening and storage operations.

8. The system according to claim 1, characterized in that The system includes a power consumption management module, which is configured to monitor the real-time workload and performance requirements of the TPU chip. When it is detected that the TPU chip is in a state of processing a complex task, the operating frequency and voltage of the TPU chip are increased; when it is detected that the TPU chip is in a low-load state or an idle state, the operating frequency and voltage of the TPU chip are reduced; and the system is also configured to collect TPU chip data when running tasks multiple times and perform statistical analysis on the data, so as to predict the task load and adjust the frequency and voltage of the TPU chip in advance.

9. The system according to claim 1, characterized in that The system includes an intelligent car, which is configured to receive the control instruction generated by the TPU chip and complete a preset action operation on the target position.

10. The system according to claim 1, characterized in that The eye-movement glasses are provided with an infrared filter.

11. An eye movement control method based on edge TPU chip model acceleration implemented by the system according to any one of claims 1 to 10, characterized in that: include: Using the eye-tracking glasses to capture external scene images and eye movement images, and transmitting them to the TPU chip via WIFI; The double-layer CNN convolution model of the TPU chip is used to receive and identify the pupil center in the eye motion image; the pupil center is mapped to the external scene image to obtain the gaze point coordinates; the Bmodel model of the TPU chip is used to receive and identify the target position coordinates in the external scene image; the gaze point coordinates and the target position coordinates are overlapped with each other; and according to the result of the coordinate overlap judgment, a control instruction is generated based on the gaze point coordinates and the target position coordinates.

Images