Eye movement tracking system

Abstract

The invention provides an eye movement tracking system. The eye movement tracking system comprises: an application processor for sending an initial control signal and a termination control signal to an image sensor, receiving a timing synchronization signal sent by the image sensor, generating a light source modulation control signal according to the timing synchronization signal, and processing image information; the light source module emits light to an eyeball according to the light source modulation control signal received from the application processor; and the image sensor senses the eyeball according to the initial control signal received from the application processor so as to generate a plurality of sensing signals, and the light source control signal enables the sensing circuit to generate a bright state sensing signal and a dark state sensing signal in a first period and a second period respectively.

Description

Technical Field

[0001] The invention relates to an eye tracking system, and in particular to an eye tracking system for obtaining images through a single light source module. Background Art

[0002] In recent years, eye tracking has been explored and applied in the three major risk industries of navigation, aviation and construction, and the application scenarios include situational type, cognitive type, emotional type, physiological type, etc. Under the popularity of the metaverse, although the application of augmented reality / virtual reality (AR / VR) glasses is relatively mature, there are several key factors that determine whether they can be adopted at the application level: first, energy consumption; second, latency; third, for event-based dynamic visual sensing (Event-Based DVS), it also includes event rate, signal-to-noise ratio (SNR), and glint detection error.

[0003] It is understandable that the power supply of AR / VR glasses comes from batteries, so the power is limited and affects the usage time of AR / VR glasses. The energy consumption of the eye tracking module mainly comes from the operation of the infrared light source and the image capture operation. The current eye tracking modules mostly use traditional camera lenses, so the image acquisition rate is at least 120 frames per second (fps), and because it uses full-screen operation (frame-to-frame), too much data is transmitted and the computing energy consumption is high. In order to solve the problem of too many data points, event-based dynamic visual sensing is also used, but in order to solve the problem of background noise, dual-light sources (dual-LED) are used when taking images. Overall, the energy consumption is still slightly high, resulting in a shortened usage time of AR / VR glasses.

[0004] In addition, some people have proposed using a hybrid event camera that combines events and standard frames, but its image acquisition rate is as high as 10,000 fps, its transmission data is large, its computing energy consumption is high, and its perception of brightness changes is weak. Therefore, under the basic condition of stabilizing the output image quality, how to reduce the energy consumption of the eye tracking system is a major issue.

[0005] Therefore, the inventors of the present application have come up with the present invention after observing the above-mentioned disadvantages. Summary of the invention

[0006] In order to overcome the disadvantages of the prior art, the present invention provides an eye tracking system, comprising: an eye tracking system, comprising: an application processor, used to send an initial control signal and a termination control signal to an image sensor, and receive a timing synchronization signal sent by the image sensor, and generate a light source modulation control signal according to the timing synchronization signal, and used to process image information; a light source module, emitting light to an eyeball according to the light source modulation control signal received from the application processor; and the image sensor, sensing the eyeball according to the initial control signal received from the application processor to generate a plurality of sensing signals, the image sensor comprising a plurality of sensing sub-pixels, the plurality of sensing sub-pixels all having an image sensing structure, wherein the image sensing structure comprises: a sensing sub-pixel; A detection circuit is exposed during a first period and a second period to generate the multiple sensing signals, wherein the duration of the first period is the same as the duration of the second period; and a storage circuit stores the multiple sensing signals including a bright state sensing signal and a dark state sensing signal; an image signal processing circuit performs calculation processing on the bright state sensing signal and the dark state sensing signal stored in the storage circuit to generate the image information, and then outputs the image information to the application processor, wherein the light source control signal enables the sensing circuit to generate the bright state sensing signal and the dark state sensing signal during the first period and the second period respectively, and wherein the application processor generates pupil movement information according to the image information, and thereafter sends the termination control signal to stop the image sensor from sensing.

[0007] Preferably, the light source module is a multi-point light source composed of a plurality of LED chips.

[0008] Preferably, the plurality of LED chips are arranged in a ring shape and include at least one of infrared light sources of two wavelength bands.

[0009] Preferably, the two wavelength bands include a band with a central wavelength of 850 nm and a band with a central wavelength of 940 nm.

[0010] Preferably, the light source modulation control signal controls the light source module to turn on an advance time from a starting point of the first period, to turn on from the starting point of the first period, or to turn on after a start-up time from the starting point of the first period, and to turn off after a shut-off time from a starting point of the second period.

[0011] Preferably, the start-up time is shorter than the duration of the first period, and the difference between the duration of the first period and the start-up time is greater than the shut-down time.

[0012] Preferably, the plurality of sensing sub-pixels can be divided into a plurality of pixel groups, wherein the plurality of pixel groups are used to sequentially output the bright-state sensing signal of the first period and the dark-state sensing signal of the second period during an imaging operation.

[0013] Preferably, the image sensor performs a first global shutter exposure operation during the first period, and performs a second global shutter exposure operation during the second period.

[0014] Preferably, the bright-state sensing signal is used to obtain image information about the pupil and the light-absorbing part of the iris, and the dark-state sensing signal is used to obtain image information about the light-reflecting point of the eyeball.

[0015] Preferably, the image sensor performs a fusion process on the bright state sensing signal and the dark state sensing signal to generate the image information.

[0016] In summary, the eye tracking system and the exposure and imaging method for eye tracking of the present invention do not need to wait until all columns (that is, complete frames) are read out before exposure fusion can be performed, but can be fused when the bright state sensing signal and the dark state sensing signal of each column are read out. Therefore, calculation and signal fusion can be performed between frames to reduce transmission data and reduce computing energy consumption. In addition, the eye tracking system generates bright state sensing signals and dark state sensing signals by using a single set of light source modules, which can reduce energy consumption while sensing the bright state sensing signals and dark state sensing signals of the eyeball, and can extend the use time of AR / VR glasses. In addition, by using a single set of light source modules, the module cost can also be reduced.

[0017] In order to make those skilled in the art understand the purpose, features and effects of the present invention, the present invention is now described in detail through the following specific embodiments and the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a signal flow chart of the eye tracking system according to the present invention;

[0019] Figure 2 is a circuit diagram of an image sensing structure according to a first embodiment of the present invention;

[0020] Figure 3 is a circuit diagram of an image sensing structure according to a second embodiment of the present invention;

[0021] Figure 4 A timing diagram of a shutter switch signal and a light emission signal according to the present invention;

[0022] Figure 5 is another timing diagram of the shutter switch signal and the light emission signal according to the present invention;

[0023] Figure 6 is another timing diagram of the shutter switch signal and the light emission signal according to the present invention;

[0024] Figure 7 A schematic diagram for explaining a sensing frame and a computing output of an image sensor according to the present invention; and

[0025] Figure 8 The figure is used to illustrate the actual image frame sensed and the calculation output of the image sensor according to the present invention.

[0026] Description of reference numerals:

[0027] 11: Application processor;

[0028] 12: light source module;

[0029] 13: Image sensor;

[0030] 21: sensing circuit;

[0031] 22: storage circuit;

[0032] 23: Amplification selection circuit;

[0033] 41: first pixel group;

[0034] 42: second pixel group;

[0035] 100: Eye tracking system;

[0036] 200, 300: image sensing structure;

[0037] 400: pixel group;

[0038] 211: sensing diode;

[0039] 212: sensing transmission circuit;

[0040] 213: sensing reset circuit;

[0041] 221: storage diode;

[0042] 222: storage transmission circuit;

[0043] 223: storage reset circuit;

[0044] 231: amplifying transistor;

[0045] 232: select transistor;

[0046] 233: signal line;

[0047] 2211: a first storage diode;

[0048] 2212: a second storage diode;

[0049] d0: start time;

[0050] d1: closing time;

[0051] d2: advance time;

[0052] EYE: pupil movement information;

[0053] FD: floating diffusion point;

[0054] FD_RST: stores reset signal;

[0055] GTX0, GTX1, GTX2, GTX3: sensing readout signal;

[0056] INF: Image information;

[0057] INI: initial control signal;

[0058] MOD: light source modulation control signal;

[0059] PD_RST: sensing reset signal;

[0060] SEL: external selection signal;

[0061] Shutter_A, Shutter_B: shutter switch signal;

[0062] SYN: timing synchronization signal;

[0063] TA: first period;

[0064] TB: second period;

[0065] TER: termination control signal;

[0066] TX0, TX1, TX2, TX3: store the read signal. DETAILED DESCRIPTION

[0067] The inventive concept will now be more fully described below with reference to the accompanying drawings in which exemplary embodiments of the inventive concept are shown. The advantages and features of the inventive concept and the methods for achieving them will become apparent from the exemplary embodiments described in more detail below with reference to the accompanying drawings. However, it should be noted that the inventive concept is not limited to the following exemplary embodiments, but can be implemented in various forms. Therefore, the exemplary embodiments are provided only to disclose the inventive concept and to make those skilled in the art understand the categories of the inventive concept. In the drawings, the exemplary embodiments of the inventive concept are not limited to the specific examples provided herein and are exaggerated for clarity.

[0068] The terms used herein are only used to illustrate specific embodiments and are not intended to limit the present invention. Unless the context clearly indicates otherwise, the singular forms of the terms "a", "an" and "the" used herein are intended to include plural forms as well. The terms "and / or" used herein include any and all combinations of one or more of the relevant listed items. It should be understood that when a component is said to be "connected" or "coupled" to another component, the component may be directly connected or coupled to the other component or there may be intermediate components.

[0069] Similarly, it should be understood that when a component (such as a layer, region, or substrate) is said to be "on" another component, the component may be directly on the other component or there may be intervening components. In contrast, the term "directly" means that there are no intervening components. It should be further understood that when the terms "include" and "comprising" are used herein, they indicate the presence of stated features, integers, steps, operations, components, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, components, and / or groups thereof.

[0070] In addition, the exemplary embodiments in the detailed description will be described by cross-sectional views that are idealized exemplary drawings of the inventive concept. Accordingly, the shapes of the exemplary drawings may be modified according to manufacturing techniques and / or tolerable errors. Therefore, the exemplary embodiments of the inventive concept are not limited to the specific shapes shown in the exemplary drawings, but may include other shapes that may be produced according to the manufacturing process. The areas illustrated in the drawings have general characteristics and are used to illustrate the specific shapes of the components. Therefore, this should not be considered as being limited to the scope of the inventive concept.

[0071] It should also be understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, these components should not be limited to these terms. These terms are only used to distinguish between various components. Therefore, the first component in some embodiments may be referred to as the second component in other embodiments without departing from the teachings of the present invention. The exemplary embodiments of the aspects of the inventive concepts explained and illustrated herein include their complementary counterparts. Throughout this specification, the same reference numerals or the same indicators represent the same components.

[0072] Furthermore, exemplary embodiments are described herein with reference to cross-sectional views and / or plan views, which are idealized exemplary illustrations. Therefore, deviations from the illustrated shapes due to, for example, manufacturing techniques and / or tolerances are expected. Therefore, exemplary embodiments should not be considered limited to the shapes of the regions shown herein, but are intended to include shape deviations due to, for example, manufacturing. Therefore, the regions shown in the figures are schematic, and their shapes are not intended to illustrate the actual shape of the regions of the device, nor are they intended to limit the scope of the exemplary embodiments.

[0073] See also Figure 1 , Figure 1 FIG. 1 is a signal flow chart of the eye tracking system 100 according to the present invention. Figure 1 As shown, the eye tracking system 100 of the present invention includes: an application processor 11, a light source module 12, and an image sensor 13. The application processor 11 is used to send an initial control signal INI and a termination control signal TER to the image sensor 13, and receive a timing synchronization signal SYN sent by the image sensor 13, and generate a light source modulation control signal MOD according to the timing synchronization signal SYN, and is used to process image information INF to generate pupil movement information EYE. The light source module 12 emits light to an eyeball according to the light source modulation control signal MOD received from the application processor 11. The image sensor 13 senses the eyeball according to the initial control signal INI and the termination control signal TER received from the application processor 11 to generate image information INF and transmit it to the application processor 11. .

[0074] Specifically, refer to Figure 1 Steps (1) to (6) shown in the figure, Figure 1The numbers in brackets represent the steps of signal transmission between components in the eye tracking system 100. First, in step (1), the eye tracking system 100 sends an initial control signal INI from the application processor 11 to the image sensor 13, so that the image sensor 13 starts sensing. Next, in step (2), the image sensor 13 sends a timing synchronization signal SYN to the application processor 11, and the application processor 11 generates a light source modulation control signal MOD according to the timing synchronization signal SYN. ​​Next, in step (3), the application processor 11 sends the light source modulation control signal MOD to the light source module 12, so that the light source module 12 emits light to the eyeball according to the light source modulation control signal MOD. Next, the image sensor 13 is used to sense the light reflected by the eyeball, so that the image sensor 13 generates image information INF. Next, in step (4), the image sensor 13 transmits the image information INF to the application processor 11, so that the application processor 11 generates pupil movement information EYE. It is understandable that in the following sensing, the signal transmission of step (2) and step (4) will be synchronized, that is, the image sensor 13 simultaneously transmits the timing synchronization signal SYN and the image information INF to the application processor 11. In addition, since the circuit design of the present invention can make the bright state sensing signal and the dark state sensing signal merge in the image sensor 13, when the image sensor 13 simultaneously transmits the timing synchronization signal SYN and the image information INF to the application processor 11, the application processor 11 can immediately send the light source modulation control signal MOD to the light source module 12 (step (3)) with almost no delay to continuously control the light emission of the light source module 12, that is, step (4), step (2), and step (3) are almost synchronized. Next, in step (5), the application processor 11 outputs the pupil movement information EYE. The above signal transmission is repeated (step (4), step (2), step (3), step (5)) until in step (6), the application processor 11 transmits the termination control signal TER to the image sensor 13 to stop the image sensor 13 from sensing. The image information INF is the image information required for pupil movement only, and the image information INF is obtained by comparing the continuous frame data obtained by the sensor after light source modulation. The pupil movement information EYE generated by the application processor 11 through the image information INF is, for example, the pupil vector, pupil area, pupil position change trajectory, etc.

[0075] In addition, through the circuit design of the image sensor 13 of the present invention, the present invention can synchronize the exposure process of the image sensor 13 with the signal processing of the image information INF. Therefore, through the dynamic visual sensing of the image sensor 13, the frame rate of the image information INF can reach 400Hz, so as to reduce the amount of calculation of the application processor 11 of the eye tracking system 100, and reduce the amount of data of the pupil movement information EYE. Therefore, the calculation energy consumption is reduced. The circuit design of the image sensor 13 of the present invention will be described below.

[0076] In addition, in some other embodiments, the application processor 11 may be an image signal processor (ISP), a central processing unit (CPU), a microprocessor (Microprocessor), a digital signal processor (DSP), a programmable logic controller (PLC), an application specific integrated circuit (ASIC), a system on chip (SoC) or other similar components or a combination of the above components.

[0077] It is understandable that the light source module 12 of the present invention may include at least one of an infrared light source module and a visible light source module. Specifically, in order to reduce unnecessary interference caused by natural light that affects the accuracy of pupil imaging, a near-infrared light source that is invisible to human vision will be used for supplementary light when using an eye tracking system. In addition, the light source may be any arrangement of a plurality of LED crystals, for example, a multi-point annular light source, for example, a multi-point light source composed of a plurality of LED crystals is densely arranged in an annular shape, parallel to the line of sight of the lens, and converged in a cone shape to form a high-brightness illumination area covering the object to be measured and eliminating dead angles. For example, there may be two circles of annular light sources arranged radially, one circle for visible light and the other for infrared light. In addition, the plurality of LED crystals may include at least one of two bands of infrared light sources, the two bands including a band with a central wavelength of 850nm and a band with a central wavelength of 940nm. However, the present invention is not limited thereto, and the arrangement, irradiation angle, type, and wavelength of the LED light source may be arbitrarily adjusted according to demand.

[0078] See also Figure 2 , Figure 22 is a circuit diagram of an image sensing structure according to a first embodiment of the present invention. In the present invention, the image sensor 13 includes a plurality of sensing sub-pixels, each of which has an image sensing structure 200. The image sensing structure 200 includes a sensing circuit 21, a storage circuit 22, and an image signal processing circuit 23. The sensing circuit 21 generates a plurality of sensing signals, the storage circuit 22 stores the plurality of sensing signals, and the image signal processing circuit 23 performs calculation processing on the sensing signals stored in the storage circuit 22 to generate image information INF, and then outputs the image information INF to the application processor 11.

[0079] Specifically, if Figure 2 As shown, the sensing circuit 21 of the present invention is mainly used to generate multiple sensing signals of different periods. In some embodiments, the sensing circuit 21 includes a sensing diode 211, a sensing transmission circuit 212 and a sensing reset circuit 213, and the cathode of the sensing diode 211 is coupled to the sensing transmission circuit 212 and the sensing reset circuit 213. Among them, the sensing diode 211 is mainly used to convert the incident light into an electric charge (that is, electrons) according to the light intensity of the incident light, the sensing transmission circuit 212 is mainly controlled by the sensing readout signals GTX0 and GTX1 to control the charge transfer between the sensing diode 211 and the storage circuit 22, and the sensing reset circuit 213 is mainly controlled by the sensing reset signal PD_RST to reset the charge stored in the sensing diode 211. It can be understood that the sensing circuit 21 of the present invention can generate multiple sensing signals of different periods by sensing the readout signal and the sensing reset signal and store them in the storage circuit 22 respectively. More specifically, as Figure 2 As shown, the sensing transmission circuit 212 of the present invention may include a plurality of metal oxide semiconductor field effect transistors (MOSFETs) to respectively receive different sensing readout signals, so that the sensing transmission circuit 212 can sequentially transmit sensing signals of different periods to the storage circuit 22 according to the sensing readout signals. Therefore, the image sensing structure 200 of the present invention can use the storage circuit 22 to store two sensing signals generated by the sensing diode 211 to the eyeball during one imaging operation. However, the present invention is not limited thereto.

[0080] Specifically, if Figure 2 As shown, the storage circuit 22 of the present invention is coupled to the sensing circuit 21, and the storage circuit 22 is used to store the above-mentioned multiple sensing signals. In some embodiments, the storage circuit 22 includes a storage diode 221, a storage transmission circuit 222, and a storage reset circuit 223, and the cathode of the storage diode 221 is coupled to the storage transmission circuit 222. Figure 2As shown, the storage diode 221 includes a first storage diode 2211 and a second storage diode 2212, and the storage transmission circuit 222 and the storage reset circuit 223 are coupled to a floating diffusion point FD; wherein the first storage diode 2211 and the second storage diode 2212 are used to store the plurality of sensing signals of different periods, and the storage transmission circuit 222 is mainly controlled by the storage readout signals TX0 and TX1 to control the charge transmission between the storage diode 221 and the floating diffusion point FD. Specifically, the storage transmission circuit 222 is coupled between the storage diode 221 and the floating diffusion point FD, and is mainly controlled by the storage readout signals TX0 and TX1 to control the charge transmission between the storage diode 221 and the floating diffusion point FD. The storage reset circuit 223 is mainly controlled by the storage reset signal FD_RST to reset the charge stored in the storage diode 221.

[0081] Specifically, Figure 2 As shown, the image signal processing circuit 23 of the present invention is coupled to the storage circuit 22. The image signal processing circuit 23 includes an amplifier transistor 231, a selection transistor 232, and a signal line 233, wherein the gate of the amplifier transistor 231 is coupled to the floating diffusion point FD, and the amplifier transistor 231 is coupled to the signal line 233 via the selection transistor 232. When the selection transistor 232 is controlled by an external selection signal SEL, so that the selection transistor 232 is in an on state, the amplifier transistor 231 amplifies the voltage of the floating diffusion point FD and transmits the sensing signal to the signal line 233. It should be particularly noted that the analog to digital converter (ADC) inside the image signal processing circuit 23 can use a differential scheme circuit. The amplifier transistor 231 in the image signal processing circuit 23 can be a first-stage amplifier transistor of an analog to digital converter, and its floating diffusion point FD is a differential input, and the other differential input is the floating diffusion point FD in the image sensing structure 200 of the adjacent row. However, the present invention is not limited thereto.

[0082] See also Figure 3 , Figure 3 FIG. 1 is a circuit diagram of an image sensing structure according to a second embodiment of the present invention. Figure 3 As shown, the sensing sub-pixels according to the second embodiment of the present invention all have an image sensing structure 300, wherein the plurality of pixel groups 400 are used to sequentially output a bright state sensing signal of a first period and a dark state sensing signal of a second period during one imaging operation. It should be further explained that, in this embodiment, the pixel group 400 may include a first pixel group 41 and a second pixel group 42, such as Figure 3As shown, the image sensing structures of the first pixel group 41 and the second pixel group 42 share the same floating diffusion point FD and the storage reset circuit 223, and the first pixel group 41 and the second pixel group 42 respectively output a bright state sensing signal during the first period and output a dark state sensing signal during the second period. However, the present invention is not limited thereto. The image sensing structures of the first pixel group 41 and the second pixel group 42 may not share the same floating diffusion point FD and the storage reset circuit 223.

[0083] It should be noted that the sensing signals of the present invention are divided into bright sensing signals and dark sensing signals. When tracking the eyeball, the bright sensing signal is the pupil image obtained when the light source is sufficient, that is, the pupil and iris light absorption information can be obtained. The dark sensing signal is the pupil image obtained when the light source is insufficient, that is, the eyeball reflective point information can be obtained. The present invention determines the exposure through the light source module 12 to obtain the bright sensing signal and the dark sensing signal.

[0084] In addition, because the bright-state sensing signal and the dark-state sensing signal are read out in an interlaced manner, it is not necessary to wait until all columns (that is, the complete frame) are read out before exposure fusion can be performed. Instead, the fusion can be performed when the bright-state sensing signal and the dark-state sensing signal of each column are finished being read out. Therefore, in the present invention, the same pixel can obtain a bright-state sensing signal and a dark-state sensing signal between frames, so as to perform calculations and signal fusion between frames, so as to reduce transmission data and reduce computing energy consumption while outputting an image with a light source spot and a clear pupil / iris margin.

[0085] It is understandable that the sensing signal is divided into a bright sensing signal in the first period and a dark sensing signal in the second period. The bright sensing signal is a signal sensed when the exposure is high, while the dark sensing signal is a signal sensed when the exposure is low. In the present invention, the exposure is determined by the light source modulation control signal MOD. That is, the light source modulation control signal MOD controls the light emission of the light source module 12 so that the sensing circuit 21 generates a bright sensing signal in the first period and generates a dark sensing signal in the second period. The method of the present invention for causing the sensing circuit 21 to generate a bright sensing signal in the first period and a dark sensing signal in the second period will be further described below. Please refer to Figure 4 , Figure 4 is a timing diagram of the shutter switch signal and the light emission signal according to the present invention. Specifically, in this embodiment, the plurality of sensing diodes 211 of the plurality of sensing sub-pixels perform a first global shutter exposure operation in the first period TA, and the plurality of sensing diodes 211 perform a second global shutter exposure operation in the second period TB. Figure 4In the embodiment, the opening and closing of the first global shutter is represented by the shutter switch signal Shutter_A, the opening and closing of the second global shutter is represented by the shutter switch signal Shutter_B, and the opening and closing of the light source module 12 is represented by the light source modulation control signal MOD. In addition, the duration of the first period TA is the same as the duration of the second period TB.

[0086] Specifically, the light source modulation control signal MOD controls the light emission of the light source module 12, so that the sensing circuit 21 generates a bright state sensing signal when the first global shutter is opened and the light source module 12 emits light in the first period TA, that is, the period of generating the bright state sensing signal is represented by the oblique black period in the shutter switch signal Shutter_A. In addition, the light source modulation control signal MOD controls the light emission of the light source module 12, so that the sensing circuit 21 generates a dark state sensing signal when the second global shutter is opened and the light source module 12 emits light in the second period TB, that is, the period of generating the dark state sensing signal is represented by the oblique black period in the shutter switch signal Shutter_B. Since the oblique black period in the shutter switch signal Shutter_A is longer than the oblique black period in the shutter switch signal Shutter_B, the sensing diode 211 performs more exposure in the first period TA, that is, generates a bright state sensing signal, and the sensing diode 211 performs less exposure in the second period TB, that is, generates a dark state sensing signal.

[0087] Further, from Figure 4 It can be seen that the light source module 12 is turned on after the start time d0 at the starting point of the first period TA, and is turned off after the close time d1 at the starting point of the second period TB. Therefore, the duration of generating the bright state sensing signal is the duration of the first period TA minus the start time d0, and the duration of generating the dark state sensing signal is the duration of the close time d1, wherein the start time d0 is less than the duration of the first period TA. Since the exposure amount of the bright state sensing signal is greater than the exposure amount of the dark state sensing signal, the oblique black period in the shutter switch signal Shutter_A is longer than the oblique black period in the shutter switch signal Shutter_B. Therefore, the difference between the duration of the first period TA and the start time d0 is greater than the close time d1.

[0088] It is understandable that the activation time d0 and the deactivation time d1 can be set arbitrarily according to requirements to adjust the sensed bright sensing signal and dark sensing signal to adjust the exposure of the bright sensing signal and the dark sensing signal according to the required pupil characteristics.

[0089] See also Figure 5 , Figure 5 FIG. 4 is another timing diagram of the shutter switch signal and the light emission signal according to the present invention. Figure 5 and Figure 4The difference is only that the start time d0 is set to zero. That is, there is no start time. When the start time d0 is set to zero, the light source modulation control signal MOD controls the light emission of the light source module 12 so that the sensing circuit 21 generates a bright state sensing signal when the first global shutter is opened and the light source module 12 emits light in the first period TA, that is, the time period for generating the bright state sensing signal is represented by the oblique black period in the shutter switch signal Shutter_A. Here, since the start time d0 is set to zero, the oblique black period is equal to the shutter switch signal Shutter_A opening period. In addition, the light source modulation control signal MOD controls the light emission of the light source module 12 so that the sensing circuit 21 generates a dark state sensing signal when the second global shutter is opened and the light source module 12 emits light in the second period TB, that is, the time period for generating the dark state sensing signal is represented by the oblique black period in the shutter switch signal Shutter_B. Therefore, it can be understood that by setting the start time d0 to zero, the opening time point of the first global shutter can be synchronized with the opening time point of the light source module 12.

[0090] See also Figure 6 , Figure 6 FIG. 4 is another timing diagram of the shutter switch signal and the light emission signal according to the present invention. Figure 6 and Figure 4 The difference is only set to be negative at the start time d0, that is, it is equivalent to setting the advance time d2. When the advance time d2 is set, the light source modulation control signal MOD controls the light source module 12 to start from the start point of the first period TA before the advance time d2, so that light can be irradiated from the light source module 12 to the eyeball within the advance time d2, so that the first global shutter can sense when it is opened.

[0091] Therefore, from the above Figure 4-6 It can be seen that compared with event-based dynamic visual sensing and traditional camera lenses, the present invention changes from a dual light source to a single light source, and can also generate pupil movement information based on the bright state sensing signal and dark state sensing signal of the eyeball. Therefore, it can reduce energy consumption and extend the use time of AR / VR glasses.

[0092] See also Figure 7 , 8 As shown, Figure 7 A schematic diagram for explaining the sensing frame and the computing output of the image sensor according to the present invention, and Figure 8 This is a diagram to illustrate the actual image frame sensed by the image sensor and the actual image output by the operation according to the present invention. Figure 7 , 8In the figure, F(x) is a frame obtained by light source modulation, wherein F(n) is a frame obtained by the first light source modulation method, i.e., a bright state sensing signal, F(n+1) is a frame obtained by the second light source modulation method, i.e., a dark state sensing signal, and P(x) is a frame obtained after the two frames F(x) are processed by calculation, i.e., a signal after image processing. As described above, the sensing signal obtained by the image sensor 13 through the sensing circuit 21 is divided into two types, for example, a bright state sensing signal and a dark state sensing signal. These two sensing signals can correspond to the frame F(n) and the frame F(n+1), respectively. Through the image signal processing circuit 23 and the storage circuit 22 of the image sensor 13, the image sensor 13 can perform calculation processing to generate a frame P(x) representing the image information INF by fusing and comparing the two frames. Then, the image sensor 13 transmits the generated image information INF (i.e., the frame P(x)) to the application processor 11. Therefore, through the circuit design of the image sensor 13 of the present invention, the present invention reduces the amount of computation of the application processor 11 of the eye tracking system 100, thereby reducing the amount of data of the pupil movement information EYE. Therefore, the computational energy consumption is reduced. Figure 8 It can be seen that the frame F(n) representing the bright state sensing signal is used to obtain the pupil and iris light absorption information, and the frame F(n+1) representing the dark state sensing signal is used to obtain the eyeball reflective point information, and the fused frame P(x) can retain the above image information while reducing the amount of calculation.

[0093] In summary, the eye tracking system and the exposure and imaging method for eye tracking of the present invention can perform exposure fusion when the bright state sensing signal and the dark state sensing signal of each column are read out instead of waiting until all the columns (that is, the complete frame) are read out. Therefore, calculation and signal fusion can be performed between frames to reduce data transmission and reduce computing energy consumption. In addition, the eye tracking system generates bright state sensing signals and dark state sensing signals by using a single set of light source modules, which can reduce energy consumption while generating pupil movement information based on the bright state sensing signals and dark state sensing signals of the eyeball, and can extend the use time of AR / VR glasses. In addition, by using a single set of light source modules, the module cost can also be reduced.

[0094] The above describes the implementation of the present invention through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of the present invention; any other equivalent changes or modifications that do not deviate from the spirit disclosed by the present invention should be included in the following patent scope.

Claims

1. An eye tracking system, characterized in that: include: An application processor, used for sending an initial control signal and a termination control signal to an image sensor, and receiving a timing synchronization signal sent by the image sensor, and generating a light source modulation control signal according to the timing synchronization signal, and used for processing image information; a light source module, emitting light to an eyeball according to the light source modulation control signal received from the application processor; and The image sensor senses the eyeball according to the initial control signal received from the application processor to generate a plurality of sensing signals. The image sensor includes a plurality of sensing sub-pixels, each of which has an image sensing structure. Wherein, the image sensing structure includes: a sensing circuit, exposed during a first period and a second period to generate the plurality of sensing signals, wherein the first period is the same as the second period; and a storage circuit for storing the plurality of sensing signals including a bright-state sensing signal and a dark-state sensing signal; and an image signal processing circuit, which performs calculation processing on the bright state sensing signal and the dark state sensing signal stored in the storage circuit to generate the image information, and then outputs the image information to the application processor, wherein the light source control signal causes the sensing circuit to generate the bright state sensing signal and the dark state sensing signal during the first period and the second period, respectively; and The application processor generates pupil movement information according to the image information, and can send out the termination control signal to stop the image sensor from sensing.

2. The eye tracking system according to claim 1, characterized in that: The light source module is a multi-point light source composed of a plurality of LED crystals.

3. The eye tracking system according to claim 2, characterized in that: The plurality of LED chips are arranged in a ring shape and include at least one of infrared light sources of two wavebands.

4. The eye tracking system according to claim 3, characterized in that: The two wavelength bands include a band with a central wavelength of 850 nm and a band with a central wavelength of 940 nm.

5. The eye tracking system according to claim 1, characterized in that: The light source modulation control signal controls the light source module to turn on from a starting point of the first period an advance time before, from the starting point of the first period, or after a start-up time from the starting point of the first period, and to turn off after a shut-off time from a starting point of the second period.

6. The eye tracking system according to claim 5, characterized in that: The start-up time is shorter than the duration of the first period, and the difference between the duration of the first period and the start-up time is longer than the shut-down time.

7. The eye tracking system according to claim 1, characterized in that: The plurality of sensing sub-pixels may be divided into a plurality of pixel groups, wherein the plurality of pixel groups are used to sequentially output the bright-state sensing signal of the first period and the dark-state sensing signal of the second period during an imaging operation.

8. The eye tracking system according to claim 1, characterized in that: The image sensor performs a first global shutter exposure operation during the first period, and performs a second global shutter exposure operation during the second period.

9. The eye tracking system according to claim 1, characterized in that: The bright state sensing signal is used to obtain image information about the pupil and the iris light absorbing part, and the dark state sensing signal is used to obtain image information about the eyeball reflective point part.

10. The eye tracking system according to claim 1, characterized in that: The image sensor performs a fusion process on the bright state sensing signal and the dark state sensing signal to generate the image information.

Images