Micro-led dynamic visual adaptation display system and method based on eye tracking

Abstract

The application discloses a Micro-LED dynamic visual adaptation display system and method based on eye movement tracking, and relates to the technical field of display driving. The system comprises a storage and calculation driving unit, a Micro-LED light emitting unit, a weight generating module and an eye movement tracking module. The storage and calculation driving unit is vertically integrated with the Micro-LED light emitting unit for integrated design. The storage and calculation driving unit is connected with the weight generating module. The weight generating module is bidirectionally electrically connected with the eye movement tracking module. The application directly embeds image enhancement operation into the internal pixel driving unit. The high-definition display of the gaze area and the low-power driving of the non-gaze area can be realized according to the gaze position of the human eye. The real-time dynamic visual adjustment adapting to the dynamic visual characteristics of the human body is achieved. The overall energy efficiency of the display system is greatly optimized while the visual experience of the user is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of display driving technology, and in particular to a Micro-LED dynamic visual adaptation display system and method based on eye tracking. Background Technology

[0002] As display technology iterates towards miniaturization, high pixel density, low power consumption, and immersive experiences, Micro-LED displays, with their core technological advantages such as high brightness, high contrast, ultra-fast response speed, long lifespan, low power consumption, and high reliability, have become a core application carrier in high-end fields such as virtual reality / augmented reality (VR / AR), wearable smart devices, automotive head-up displays, and portable high-definition displays. These high-end applications place increasingly stringent demands on the dynamic visual adaptation and low-power operation of display technologies. The human visual system exhibits significant dynamic characteristics: the fovea region of the human eye has an extremely high ability to distinguish visual details, requiring high-definition, high-contrast high-definition displays only for the focused area, while the visual sensitivity of the non-focused areas in the peripheral region of the retina is significantly reduced, requiring no high-parameter displays. Therefore, eye-tracking-based dynamic visual adaptation technology has become a core development direction for improving the immersive experience of high-end Micro-LED display products and reducing system power consumption, and is also a key research focus and technical challenge in the industry.

[0003] In the existing technology system, the eye-tracking and dynamic visual adaptation solutions of Micro-LED display technology all adopt an external computing architecture that separates computing, storage, driving, and display. Although this traditional solution can achieve basic eye-tracking and visual adaptation functions, it suffers from drawbacks such as significant response delay, large loss of computing accuracy, high system power consumption, complex hardware architecture, and many difficult-to-overcome technical bottlenecks due to the limited separate architecture design and process arrangement. It can no longer meet the development needs of high-end Micro-LED display applications. Therefore, for those skilled in the art, how to design a device-level eye-tracking dynamic visual adaptation solution that integrates storage, computing, and display is an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a Micro-LED dynamic visual adaptation display system and method based on eye tracking to solve the problems mentioned in the background technology. It can innovate from the hardware architecture and method flow level to achieve pixel-level fusion of computing, driving and display. While improving the real-time performance and accuracy of visual adaptation, it can effectively reduce system power consumption and hardware complexity, and adapt to the core development needs of Micro-LED display technology such as miniaturization, high pixel density, low power consumption and immersive experience.

[0005] To achieve the above objectives, the present invention provides the following solution: On one hand, it provides a Micro-LED dynamic visual adaptation display system based on eye tracking, including a memory-based driving unit, a Micro-LED light-emitting unit, a weight generation module, and an eye tracking module; wherein, the memory-based driving unit and the Micro-LED light-emitting unit are vertically integrated into a single design, the memory-based driving unit is connected to the weight generation module, and the weight generation module is bidirectionally electrically connected to the eye tracking module.

[0006] Preferably, the memory-based driving unit includes a gating transistor and a floating-gate memory transistor; the gate of the gating transistor is connected to a row gating signal, the drain is connected to a column gating signal, and the source is electrically connected to the gate of the floating-gate memory transistor; the drain of the floating-gate memory transistor is electrically connected to the column gating signal, and the source is electrically connected to the positive terminal of the Micro-LED light-emitting unit; the memory-based driving unit is used for weight calculation and driving the display.

[0007] By adopting the above technical solution, the following beneficial technical effects are achieved: the floating gate storage transistor and the gating transistor are integrated to form a storage-computing driving unit. By fully relying on the three core characteristics of the floating gate storage transistor—in-situ multiplication and addition, precise adjustment of threshold voltage, and non-volatile charge storage—and combining the collaborative work of the eye-tracking module and the weight generation module, the real-time capture of human eye visual coordinates and the dynamic adjustment of pixel-level operator weights on the display panel can be realized.

[0008] Preferably, the eye-tracking module is a non-contact optical sensing module, including an infrared imaging sensor and an eye-tracking recognition algorithm chip; the eye-tracking module is used to identify and extract eye-tracking data, and transmit the eye-tracking data to the weight generation module in real time.

[0009] By adopting the above technical solution, the following beneficial technical effects are achieved: This invention uses a non-contact high-precision eye-tracking module to achieve high-frame-rate accurate capture of human eye gaze coordinates and motion trajectories. Combined with the dynamic region division logic of the weight generation module, it can intelligently divide the gaze or non-gaze regions according to the real-time focusing position of the human eye, and match customized operator weights for different regions, so that the display effect is highly consistent with human visual perception, greatly improving the user's immersive visual experience, adapting to the core needs of immersive display scenarios such as VR / AR, providing a good visual experience, accurately adapting to the dynamic visual characteristics of the human eye, and achieving an immersive display effect.

[0010] Preferably, the weight generation module is used to receive eye movement data from the eye tracking module, dynamically divide the gaze area and non-gaze area of ​​the display panel according to the real-time human eye gaze coordinates of the eye movement data, and generate different weight operators to transmit to the storage and computing drive unit in real time.

[0011] Preferably, the in-memory computing driving units are arranged in an array in the display panel, and each in-memory computing driving unit and the corresponding Micro-LED light-emitting unit are integrated into a vertically integrated structure using thermo-press bonding.

[0012] Preferably, the eye movement data includes panel coordinates of eye gaze, eye movement trajectory, and gaze focus range.

[0013] Preferably, the weight generation module generates dynamic operator weights and writes them in real time into the floating gate layer of the floating gate memory transistor of the corresponding pixel through charge injection processes such as hot electron injection or tunneling injection, thereby realizing pixel-level loading of operator weights. Furthermore, the module completes local or global refresh of operator weights based on the real-time changes in eye coordinates output by the eye tracking module.

[0014] On the other hand, a Micro-LED dynamic visual adaptation display method based on eye tracking is provided, which utilizes the aforementioned Micro-LED dynamic visual adaptation display system based on eye tracking for adaptation display. The specific steps include the following:

[0015] The eye-tracking module is used to capture the visual coordinates of the human eye in real time.

[0016] In the weight generation module, weights are allocated in real time based on the human eye visual coordinates and transmitted to the storage and computing drive unit.

[0017] The in-memory computing drive unit performs weight calculations and converts the calculation results into drive currents to drive the Micro-LED light-emitting units, thereby completing dynamic visual adaptation display.

[0018] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0019] (1) Eye-tracking technology is deeply integrated with Micro-LED storage, computing and display integrated drive architecture. Through innovative hardware structure design and integrated optimization of process flow, eye-tracking capture, weight control, in-situ computing and dynamic display are achieved at the pixel level. Compared with existing technologies, dynamic visual adaptation and system energy efficiency are both breakthroughs.

[0020] (2) The calculation process of image enhancement and weight adjustment is directly embedded in the pixel-level storage and computing drive unit, which abandons the traditional solution of "external calculation - data transmission - display drive" separation process and avoids the double delay caused by data cross-module transmission and external digital calculation. At the same time, the weight generation module can complete the local / global real-time refresh of operator weights according to eye movement changes, and the floating gate storage and computing transistor realizes the fast writing and calculation of weights. The whole realizes millisecond-level eye movement-display adaptation response, solves the delay problem of traditional solutions, and ensures the real-time performance of dynamic visual adaptation.

[0021] (3) The system has low power consumption and significantly improved energy efficiency, which meets the needs of low power display scenarios; the calculation accuracy is high and the display effect is excellent. This invention utilizes the intrinsic electrical characteristics such as the transfer curve of the floating gate memory transistor to complete the point-to-point multiplication and addition operation of the input current signal and the operator weight in the analog domain. No digital signal conversion is required throughout the process, avoiding the rounding error in the traditional digital calculation process and the accuracy loss caused by attenuation and crosstalk in the signal transmission process; and the calculation result is directly used as the driving current to drive the Micro-LED light-emitting unit, realizing pixel-level precise driving, greatly improving the clarity, contrast and detail of the display screen, and ensuring high-definition display effect. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a diagram showing the connection relationship between the memory-based driving unit and the Micro-LED light-emitting unit of the present invention;

[0024] Figure 2 This is a schematic diagram of the weighting operation principle of the floating gate memory transistor of the present invention;

[0025] Figure 3 This is a flowchart of the method of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] The purpose of this invention is to provide a Micro-LED dynamic visual adaptation display system based on eye tracking, including a memory-based driving unit, a Micro-LED light-emitting unit, a weight generation module, and an eye tracking module. The memory-based driving units are arranged in an array in the display panel. Each memory-based driving unit and its corresponding Micro-LED light-emitting unit are integrated into a vertically integrated structure using thermo-press bonding. The memory-based driving units are connected to the weight generation module, and the weight generation module is connected to the eye tracking module via bidirectional electrical signals. The above modules work together to complete the integrated functions of eye tracking, weight generation, pixel-level calculation, and dynamic display.

[0028] The in-memory computing driver unit includes a gating transistor and a floating-gate in-memory transistor, which work together to achieve pixel gating, charge storage, threshold control, in-situ multiply-accumulate operations, and current driving; the connection between the in-memory computing driver unit and the Micro-LED light-emitting unit is as follows: Figure 1 As shown, the gating transistor serves as the core of gating control for the entire display array. Its gate is connected to the row gating signal, its drain to the column gating signal, and its source is electrically connected to the gate of the floating-gate memory transistor (FMT). This allows for precise directional transmission of the gating signal, while simultaneously controlling the row and column gating of pixel units and isolating signals between adjacent pixels to avoid crosstalk. The FMT integrates three core functions: charge storage, precisely adjustable threshold voltage, and in-situ multiplication-addition. Its drain is electrically connected to the column gating signal, its source to the positive terminal of the Micro-LED light-emitting unit, and the negative terminal of the Micro-LED light-emitting unit is connected to the row line, thus forming a closed and independent pixel-level integrated "computation-driving-display" loop. The FMT can adjust its threshold voltage according to the written operator weights using charge injection processes such as hot electron injection or tunneling injection. Utilizing its intrinsic electrical characteristics, such as its transfer curve, it performs point-to-point multiplication-addition of the input electrical signal and operator weights in the analog domain. The result is directly converted into a driving current to drive the Micro-LED light-emitting unit.

[0029] Furthermore, the eye-tracking module is a non-contact, high-precision optical sensing module, including an infrared imaging sensor and an eye-tracking recognition algorithm chip. It can continuously scan and capture the user's eye state at a high frame rate, accurately identify and extract eye-tracking data such as eye gaze coordinates, eye movement trajectory, and gaze focus range. It can also transmit the collected eye-tracking data to the weight generation module in real time without delay, providing an accurate data source for subsequent weight allocation.

[0030] Furthermore, the weight generation module is a dedicated control module integrating data processing and signal output. It incorporates visual characteristic algorithms and pixel-level weight allocation logic. It receives eye-tracking data from the eye-tracking module, dynamically divides the viewing area and non-viewing area of ​​the display panel based on the real-time eye gaze coordinates of the eye-tracking data, and generates different weight operators that are transmitted in real-time to the storage-and-computation drive unit. For example... Figure 2 As shown, the weight generation module dynamically divides the viewing area (the core area currently focused by the human eye) and the non-viewing area (the peripheral area not focused by the human eye) of the display panel; it generates high-weight operators (Vth) with high definition and high contrast for the viewing area, and generates low-weight operators (Vth) that meet visual perception requirements and have low power consumption for the non-viewing area. It can also transmit the generated pixel-level dynamic operator weights in real time to the floating gate memory transistors of the corresponding pixel-level memory drive unit in the form of electrical signals, so as to achieve accurate loading of weights.

[0031] Specifically, the weight generation module uses FPGA / ASIC / MCU as the core processing unit, and is configured with parameters such as computing power and clock frequency to meet the requirements of millisecond-level pixel weight calculation; it uses LVDS / SPI interface to communicate with the eye tracking module, and uses high-speed differential interface to communicate with the in-memory computing drive array, defining the corresponding communication protocol, data transmission rate and timing synchronization mechanism; and it is configured with data buffer, digital-to-analog converter (DAC) and signal conditioning unit.

[0032] The weight generation module generates different weight operators using the following methods: Kalman filtering / sliding window filtering algorithms are used to denoise and calibrate the eye-tracking data, and effective gaze determination thresholds such as pixel offset and eye movement speed are set; the gaze region is defined based on Gaussian / circular / elliptical models, and parameters such as region radius and gradient decay coefficient are set, with gaze offset of N pixels used as the trigger condition for dynamic region update; the quantization mapping formula / curve between the weight operator (Vth) and brightness / driving current is defined, the numerical range of high weight in the gaze region and low weight in the non-gaze region is clarified, and pixel-level weight gradient decay rules matching the distance to the gaze point are formulated, and the corresponding weights are written into the in-memory transistor. The signal processing flow is as follows: the eye-tracking data transmitted by the eye-tracking module is cached in hardware, and data denoising and trajectory prediction are completed through software algorithms to output effective human eye gaze coordinates; the main control unit calls the region division algorithm to generate gaze / non-gaze region masks of the display panel in real time; based on the region masks, the pixel weights of the entire array are calculated in parallel / serial mode, and the conversion and encoding of weight values ​​into electrical signals are completed; the weight data is transmitted to the in-memory driving array through a high-speed interface, and CRC check is used to ensure the integrity of data transmission.

[0033] The weight generation module establishes a hardware synchronization clock with the eye tracking module to ensure that the timing of eye tracking data acquisition and weight generation is consistent; it defines the hardware trigger signal and refresh range determination rules for local / global weight refresh with the in-memory computing drive array, and configures the weight writing voltage and timing parameters adapted to the hot electron / tunneling injection process; it coordinates the timing of weight signal output and row and column gating signals with the display array to achieve synchronous matching of weight loading and pixel driving.

[0034] On the other hand, based on the aforementioned dynamic visual adaptation display system, a Micro-LED dynamic visual adaptation display method based on eye tracking is proposed. This method abandons the external calculation and data transmission processes of traditional solutions, directly relying on the physical characteristics of the device to complete real-time calculation and driving of the analog domain within the pixel, achieving precise adaptation to the dynamic visual characteristics of the human body, such as... Figure 3 As shown, the specific steps include the following:

[0035] S1. Real-time capture of human eye visual coordinates using an eye-tracking module;

[0036] S2. In the weight generation module, weights are allocated in real time based on human visual coordinates and transmitted to the storage-driven unit.

[0037] S3. The storage-based driving unit performs weight calculations and converts the calculation results into driving current to drive the Micro-LED light-emitting unit, thus completing the dynamic visual adaptation display.

[0038] Furthermore, in S1, the eye-tracking module is the main execution unit, which continuously scans the user's eye state at a high frame rate. Through infrared imaging and algorithm recognition, it accurately captures eye movement data such as the panel coordinates of the human eye gaze, the trajectory of the eye movement, and the range of the gaze focus. After real-time preprocessing of the collected eye movement data, it is transmitted to the weight generation module in a delay-free manner, providing an accurate basis for weight allocation.

[0039] In S2, the weight generation module is the main execution unit. Based on the received real-time eye-tracking data, it dynamically divides the gaze area and non-gaze area of ​​the display panel through a built-in algorithm. According to the dynamic visual characteristics of the human body, high operator weights are assigned to the gaze area and low operator weights are assigned to the non-gaze area. A unique matching operator weight is generated for each pixel unit of the display panel to form full-panel pixel-level weight distribution data.

[0040] In S3, the generated pixel-level dynamic operator weights are written in real-time and precisely into the floating gate layer of the corresponding pixel's floating gate memory transistor using a charge injection process involving hot electron injection or tunneling injection. This achieves pixel-level precise loading of operator weights and allows for local or global updates of operator weights based on real-time changes in eye-tracking coordinates, ensuring real-time weight adjustment. Using the pixel-level memory driver unit as the execution entity, the original image data is converted into an input current signal Vin via digital-to-analog conversion and input to the pixel-level memory driver unit. The floating gate memory transistor performs point-to-point in-situ multiplication and addition operations between the input current signal Vin and the operator weight Vth within the pixel. The result Iout is directly used as the driving current input to the Micro-LED light-emitting unit. The gaze area achieves high-definition, high-contrast, and high-detail high-definition display due to the loading of high-weight operators, while the non-gaze area achieves low-current and low-power energy-saving driving due to the loading of low-weight operators. Ultimately, this achieves real-time dynamic display adapted to the dynamic visual characteristics of the human body, realizing the effect of "high-definition display at the visual focus point and low-power operation at the visual non-focus point."

[0041] In this invention, the Micro-LED display array adopts an architecture design of row and column gating collaborative control and pixel-level dynamic weight refresh, realizing precise control of individual pixels and consistency and real-time guarantee of dynamic visual adaptation of the entire array under large-scale array:

[0042] Row and column gating control: The row gating signal adopts a bus design, which is electrically connected to the gate of all gating transistors in the display array to realize synchronous gating control of the entire row of pixels; the column gating signal also adopts a bus design, which is electrically connected to the drain of all gating transistors and floating gate memory transistors in the corresponding column to realize precise input of column-level signals; through the combined on / off control of row and column gating signals, it is possible to accurately locate and drive a single pixel-level memory drive unit, and realize independent calculation, driving and display control of a single pixel.

[0043] Dynamic weight coordination: The weight generation module establishes a high-speed real-time signal connection with the array-type pixel-level storage and computing drive unit. Based on the eye movement coordinate changes captured by the eye-tracking module, it can locally refresh (only refresh the boundary pixels between the gaze area and the non-gaze area) or globally refresh (update the pixel weights of the entire panel) the operator weights of the entire display array. This ensures the millisecond-level response speed of weight adjustment and the consistency of the entire array display, achieving zero-delay adaptation between eye movement and display. Ultimately, it achieves efficient and accurate dynamic visual adaptation display under large-scale arrays.

[0044] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A Micro-LED dynamic visual adaptation display system based on eye tracking, characterized in that, It includes a memory-based computing drive unit, a Micro-LED light-emitting unit, a weight generation module, and an eye-tracking module; wherein, the memory-based computing drive unit and the Micro-LED light-emitting unit are vertically integrated into a single design, the memory-based computing drive unit is connected to the weight generation module, and the weight generation module is bidirectionally electrically connected to the eye-tracking module.

2. The Micro-LED dynamic visual adaptation display system based on eye tracking according to claim 1, characterized in that, The in-memory computing driving unit includes a gating transistor and a floating-gate in-memory transistor; the gate of the gating transistor is connected to a row gating signal, the drain is connected to a column gating signal, and the source is electrically connected to the gate of the floating-gate in-memory transistor; the drain of the floating-gate in-memory transistor is electrically connected to the column gating signal, and the source is electrically connected to the positive terminal of the Micro-LED light-emitting unit; the in-memory computing driving unit is used for weight calculation and driving the display.

3. The Micro-LED dynamic visual adaptation display system based on eye tracking according to claim 1, characterized in that, The eye-tracking module is a non-contact optical sensing module, including an infrared imaging sensor and an eye-tracking recognition algorithm chip; the eye-tracking module is used to identify and extract eye-tracking data, and transmit the eye-tracking data to the weight generation module in real time.

4. The Micro-LED dynamic visual adaptation display system based on eye tracking according to claim 1, characterized in that, The weight generation module is used to receive eye movement data from the eye tracking module, dynamically divide the gaze area and non-gaze area of ​​the display panel according to the real-time human eye gaze coordinates of the eye movement data, and generate different weight operators to transmit to the storage and computing drive unit in real time.

5. The Micro-LED dynamic visual adaptation display system based on eye tracking according to claim 1, characterized in that, The in-memory computing driving units are arranged in an array in the display panel, and each in-memory computing driving unit and the corresponding Micro-LED light-emitting unit are integrated into a vertically integrated structure using thermo-press bonding.

6. The Micro-LED dynamic visual adaptation display system based on eye tracking according to claim 3, characterized in that, The eye-tracking data includes panel coordinates of eye gaze, eye movement trajectory, and gaze focus range.

7. A Micro-LED dynamic visual adaptation display system based on eye tracking according to claim 2, characterized in that, The weight generation module generates dynamic operator weights, which are written in real time into the floating gate layer of the floating gate memory transistor of the corresponding pixel through charge injection processes such as hot electron injection or tunneling injection, thereby realizing pixel-level loading of operator weights. Furthermore, the module completes local or global refresh of operator weights based on the real-time changes in eye coordinates output by the eye tracking module.

8. A Micro-LED dynamic visual adaptation display method based on eye tracking, characterized in that, The adaptive display using the Micro-LED dynamic visual adaptation display system based on eye tracking as described in any one of claims 1-7 includes the following specific steps: The eye-tracking module is used to capture the visual coordinates of the human eye in real time. In the weight generation module, weights are allocated in real time based on the human eye visual coordinates and transmitted to the storage and computing drive unit. The in-memory computing drive unit performs weight calculations and converts the calculation results into drive currents to drive the Micro-LED light-emitting units, thereby completing dynamic visual adaptation display.

Images