Eye tracking sensor

Abstract

An eye tracking sensor of the present invention comprises a pixel array consisting of an MХN cellular neural network having M pixel columns and N pixel rows, wherein the pixel array comprises: a capturing unit for capturing image data of an eye image for each pixel; and a processing unit for acquiring binary data for each pixel of the captured image data, and performing expansion for converting a white area of the eye image into a black pixel and contraction for converting a black area of the eye image into a white pixel, on the basis of binary data of a predetermined number of peripheral pixels for the binary data for each pixel, so as to provide information on the position of a reflection caused by external light in the eye image.

Description

eye tracking sensor

[0001] The present invention relates to a gaze tracking sensor, and more particularly, to a gaze tracking sensor implemented as a single chip that tracks a user's gaze by finding the pupil position and the reflection position due to external light in an eye image through pixel-level image processing.

[0002] Eye tracking, or eye tracking, is a method of tracking the location or movement of a user's gaze. It is widely used in human-computer interaction (HCI) such as virtual reality (VR), and its utilization is gradually increasing in various fields such as marketing, medicine, and rehabilitation.

[0003] In this type of eye tracking, a method is often used in which an eye image is captured by an image sensor and an image processing algorithm is built into a separate DSP (digital signal processing chip) or microprocessor to perform image processing on the captured eye image.

[0004] However, DSP chips require a lot of power consumption, and due to performance limitations of DSP chips for image processing, it sometimes takes a long time to detect accurate gaze information, such as pupil position.

[0005] Meanwhile, in order to accurately identify the user's gaze direction as gaze information, there are cases where infrared rays are irradiated to the eye in addition to the pupil position in the eye image to further detect the reflection position reflected by the retina. In some cases, a method is used in which these reflection positions are detected by a separate component such as an IR position sensitive detector (PSD).

[0006] However, in the case of reflection location detection using these separate components, there was a problem that it was difficult to reduce the size of the product as the number of components increased, and the cost of components and PCB (printed circuit board) manufacturing also increased.

[0007] The present invention is intended to solve the above-mentioned problem and provides a gaze tracking sensor capable of detecting both pupil position and reflection position with a single chip.

[0008] According to one embodiment, an eye tracking sensor includes a pixel array composed of an MХN cell neural network having M pixel columns and N pixel rows, wherein the pixel array includes: a capture unit that captures image data of an eye image for each pixel; and a processing unit that obtains binary data for each pixel of the captured image data, and performs expansion for converting a white area in the eye image into a black pixel based on binary data of a predetermined number of surrounding pixels for each binary data for each pixel, and contraction for converting a black area in the eye image into a white pixel, so that information on a reflection position due to external light in the eye image is provided.

[0009] The pixel array includes an inverter that selectively inverts binary data for each pixel, and the processing unit can perform expansion and contraction on the binary data for each pixel inverted through the inverter, thereby providing information on the pupil position in the eye image.

[0010] The processing unit may include an expansion means for performing expansion to convert the target pixel in the white area to black when at least one of the surrounding pixels is black; and a shrinking means for performing shrinking to convert the target pixel in the black area to white when at least one of the surrounding pixels is white.

[0011] The method may further include a termination decision unit that sums binary data for each pixel based on the output value of the processing unit according to expansion and contraction for the entire pixel column and pixel row, and terminates expansion and contraction when the number of black pixels according to the sum becomes smaller than a predetermined reference number.

[0012] When expansion and contraction are completed, the processing unit may further include a column selection unit that selects a column position of one of a plurality of black pixels, and a row selection unit that selects a row position of one of a plurality of black pixels, based on the final output value of the processing unit.

[0013] Meanwhile, a gaze tracking method using an eye tracking sensor including a pixel array composed of an M×N cellular neural network having M pixel columns and N pixel rows includes: an operation of capturing image data of an eye image for each pixel; an operation of obtaining binary data for each pixel of the captured image data; an operation of performing expansion for converting a white area in the eye image into a black pixel and a contraction for converting a black area in the eye image into a white pixel based on binary data of a predetermined number of surrounding pixels for the binary data for each pixel; and an operation of providing information on a pupil position and a reflection position due to external light in the eye image.

[0014] According to one embodiment of the present invention as described above, by providing a gaze tracking sensor capable of detecting both pupil position and reflection position, there is an effect of quickly obtaining more accurate gaze information, including the user's gaze direction.

[0015] Additionally, by implementing the eye tracking sensor as a single chip, the number of parts is minimized, making it easy to miniaturize the product and reducing manufacturing costs.

[0016] Additionally, by implementing each component of the eye tracking sensor as a digital circuit, it is possible to reduce power consumption and miniaturize the chip more easily.

[0017] FIG. 1 is a block diagram illustrating the configuration of a gaze tracking sensor according to one embodiment of the present invention.

[0018] FIG. 2 illustrates a circuit diagram of a pixel array according to one embodiment of the present invention.

[0019] FIG. 3 and FIG. 4 are diagrams illustrating acquisition of binary data of an eye image for detecting pupil position and reflection position, respectively, in a pixel array according to one embodiment of the present invention.

[0020] Figures 5 and 6 are drawings each illustrating the expansion and contraction process of a pixel array according to an embodiment of the present invention.

[0021] FIG. 7 and FIG. 8 are diagrams for summing binary data and determining the end point in a pixel array according to one embodiment of the present invention.

[0022] FIG. 9 is a drawing regarding determination of a pupil position or reflection position in a pixel array according to one embodiment of the present invention.

[0023] Figure 10 is a flowchart of an operation for tracking a pupil position or a reflection position in a gaze tracking sensor according to one embodiment of the present invention.

[0024] FIG. 11 and FIG. 12 are drawings each showing an example of tracking the pupil position and reflection position in an eye image according to one embodiment of the present invention.

[0025] Fig. 13 illustrates a circuit diagram of a pixel array according to another embodiment of the present invention.

[0026] FIG. 14 and FIG. 15 are drawings showing the configuration of a circuit chip module of a gaze tracking sensor according to another embodiment of the present invention.

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that a person having ordinary skill in the art to which the present invention pertains can easily practice the present invention.

[0028] Fig. 1 is a block diagram illustrating the configuration of a gaze tracking sensor according to one embodiment of the present invention. Fig. 2 illustrates a circuit diagram of a pixel array according to one embodiment of the present invention.

[0029] A gaze tracking sensor (1) is provided in an electronic device such as a head-mounted display apparatus (HMD) or smart glasses, for example, and provides information on the user's gaze.

[0030] According to one embodiment, the eye tracking sensor (1) is implemented as a single-chip complementary metal oxide silicon (CMOS) image sensor (CIS: CMOS image sensor). The CMOS image sensor is an image sensor manufactured using CMOS manufacturing technology, and since circuits for image signal processing can be integrated into a single chip, it is easier to miniaturize the module through a SoC (System On Chip). In addition, the CMOS image sensor enables pixel-level image processing by a pixel array (100) described below.

[0031] According to one embodiment of the present invention, the gaze tracking sensor (1) can perform two functions. First, the gaze tracking sensor (1) performs a function of providing gaze information of a user based on captured image data, i.e., an eye image. Second, the gaze tracking sensor (1) performs a function of outputting the captured image data through an image output unit (400). In one embodiment, the image output unit (400) includes a CDS (Correlated Double Sampling) circuit and an A / D converter (ADC), and can convert the captured image data into a digital signal and output it.

[0032] According to one embodiment, the gaze tracking sensor (1) can operate in two modes: a pupil position tracking mode that tracks the position of the pupil (eye) in an eye image, and a reflection position tracking mode that tracks the position of a reflection (glint) of external light such as infrared light. That is, the gaze tracking sensor (1) according to the present invention tracks the reflection position, thereby enabling an electronic device equipped with the gaze tracking sensor (1) to more accurately and quickly identify the gaze direction of a user.

[0033] The eye tracking sensor (1) can selectively operate in, for example, one of a pupil position tracking mode and a reflection position tracking mode, and can provide information on the pupil position or the reflection position corresponding to each operation mode. That is, according to one embodiment of the present invention, the eye tracking sensor (1) is implemented so as to be capable of detecting (tracking) both the pupil position and the reflection position in image data.

[0034] The gaze tracking sensor (1) includes a pixel array (100).

[0035] Referring to FIG. 1, a pixel array (100) is composed of an M×N cellular neural network having M pixel columns (hereinafter, also referred to as columns) and N pixel rows (hereinafter, also referred to as rows). Each pixel of the pixel array (100) outputs a value representing its own state, for example, a black pixel or a white pixel, and the output of each pixel can be summed for each entire column and row.

[0036] The pixel array (100) includes a capture unit (not shown) that captures image data of the eye image for each pixel by capturing an image of the user's eye. The capture unit may be implemented as, for example, an array-type capture assembly, so that image data for each pixel can be captured by each capture unit of the capture assembly.

[0037] In one embodiment, the capture unit may be configured to capture image data pixel by pixel in a global shutter manner. However, this is not limited to the above, and as another example, the capture unit may be configured in a rolling shutter manner.

[0038] According to one embodiment, the capture unit can capture image data (e.g., the second image data of FIG. 4) while irradiating external light (e.g., infrared light) to the user's eyes. When external light is irradiated to the user's eyes, the light is reflected by the retina of the eye, causing the pupil to brighten (Bright-Eye), and the location where the external light is reflected (glint) from the cornea is detected. Using the positional relationship between these two parts, the gaze direction of the eyes can be tracked. For example, when capturing image data for each pixel while irradiating infrared light, the phenomenon of the pupil brightening and the reflection location are detected, and when the eyes move up, down, left, and right, the gaze direction can be tracked using the relative positional relationship between the changing center point (pupil location) of the brightened pupil and the reflection location.

[0039] According to one embodiment, the pixel array (100) provides information on the pupil position in captured image data, i.e., an eye image, or information on the reflection position by external light such as infrared rays. For example, the pixel array (100) can output information on the pupil position or information on the reflection position corresponding to either the pupil position tracking mode or the reflection position tracking mode.

[0040] Referring to FIG. 2, the pixel array (100) includes a binarization unit (110) and a processing unit (120) for each pixel.

[0041] According to one embodiment, the binarization unit (110) and the processing unit (120) are connected to each capture unit and can process image data for each pixel.

[0042] FIG. 3 and FIG. 4 are diagrams illustrating acquisition of binary data of an eye image for detecting pupil position and reflection position, respectively, in a pixel array according to one embodiment of the present invention.

[0043] Referring to FIG. 2, the binarization unit (110) includes a comparator (111) that compares pixel data of each pixel of captured image data with a threshold value and outputs binary data.

[0044] According to one embodiment, the binarization unit (110) includes an inverter (112), that is, a NOT gate, which inverts binary data output from the comparator (111). Here, the inverter (112) can selectively invert binary data output from the comparator (111). In addition, the binarization unit (110) may include a buffer gate (113) which outputs binary data output from the comparator (111) as is.

[0045] According to one embodiment, binary data output from the comparator (111) is input to either the inverter (112) or the buffer gate (113). For example, in the pupil position tracking mode, binary data may be input to the inverter (112), and in the reflection position tracking mode, binary data may be input to the buffer gate (113). Here, the binarization unit (110) receives an external signal indicating the pupil position tracking mode or the reflection position tracking mode, and in response, binary data may be input to either the inverter (112) or the buffer gate (113). However, this is merely an example, and may be implemented in a form in which binary data is input to the buffer gate (113) in the pupil position tracking mode, and is input to the inverter (112) in the reflection position tracking mode. In addition, as another embodiment of the pixel array (100), a bypass path for the inverter (112) may be provided for each pixel without a buffer gate.

[0046] In the following, an example will be described in which binary data is input to an inverter (112) when tracking pupil position and binary data is input to a buffer gate (113) when tracking reflection position.

[0047] Referring to FIGS. 3 and 4, the comparator (111) can compare pixel data (pixel input) of each pixel of captured image data with a predetermined threshold value (external threshold), and output binary data corresponding to the comparison result, for example, 1 or 0.

[0048] In one embodiment, the comparator (111) outputs low, i.e., 0, when the pixel data is below the threshold value, and outputs high, i.e., 1, when the pixel data is greater than the threshold value. Here, 0 is defined as a value representing a black pixel, and 1 is defined as a value representing a white pixel, respectively.

[0049] Referring to FIG. 3, it can be confirmed that the image data of the eye image captured in the pupil position tracking mode (hereinafter also referred to as first image data) has pixel data (pixel values) of pixels in the first area (PA1) corresponding to the pupil position having relatively larger values ​​than other pixels.

[0050] Each pixel data of the first image data is converted into binary data of 1 or 0 through a comparator (111) and inverted through an inverter (112). Accordingly, as shown in FIG. 3, the binary data (binarized pixel output) of the first image data may be converted into 0 (Px.OUT = 0) for pixel data of the first area (PA1) having a value greater than the threshold value, and converted into 1 (Px.OUT = 1) for pixel data of the remaining areas. The binarization unit (110) outputs binary data of the first image data converted through the comparator (111) and the inverter (112) as described above.

[0051] Referring to FIG. 4, it can be confirmed that the image data of the eye image captured in the reflection position tracking mode (hereinafter also referred to as second image data) has pixel data (pixel values) of pixels in the second area (PA2) corresponding to the reflection position having relatively smaller values ​​than other pixels.

[0052] Each pixel data of the second image data is converted into binary data of 1 or 0 through the comparator (111) and passes through the buffer gate (113). Accordingly, as shown in FIG. 4, the binary data (binarized pixel output) of the second image data may be converted into 0 (Px.OUT = 0) for the pixel data of the second area (PA1) having a value lower than or equal to the threshold value, and may be converted into 1 (Px.OUT = 1) for the pixel data of the remaining areas. The binarization unit (110) outputs the binary data of the second image data converted through the comparator (111) and the buffer gate (113) as described above.

[0053] According to one embodiment of the present invention as described above, the first image data and the second image data provide binary data that are inverted from each other through the binarization unit (110).

[0054] For example, in the first image data, pixel data in the first area (PA1) that has a value greater than the threshold value can be converted to black pixels (0), and the remaining pixel data can be converted to white pixels (1). In contrast, in the second image data, pixel data in the second area (PA2) that has a value less than the threshold value can be converted to black pixels (0), and the remaining pixel data can be converted to white pixels (0).

[0055] The processing unit (120) processes the captured image data.

[0056] According to one embodiment, the processing unit (120) is configured to process binary data for each pixel output from the binarization unit (110), and specifically, can perform expansion to convert a white area in the eye image into a black pixel (0), or shrink to convert a black area in the eye image into a white pixel (1).

[0057] The eye tracking sensor (1) according to one embodiment of the present invention can be configured to perform expansion and contraction for each pixel of the pixel array (100) as described above, and to identify the pupil position and reflection position through such pixel-level interaction.

[0058] Referring to FIG. 2, the processing unit (120) includes a first multiplexer (121), a flip-flop unit (DFF) (122), an expansion means (All-1 Detector) (123), a contraction means (All-0 Detector) (124), and a second multiplexer (125).

[0059] The first multiplexer (121) uses a selection signal (sel) to transmit binary data for each pixel output from the binarization unit (110) to the flip-flop unit (122).

[0060] The flip-flop unit (122) is implemented as a D flip-flop and stores binary data output from the first multiplexer (121) in synchronization with the clock signal (CLK). Here, the flip-flop unit (122) can store binary data for one clock. The binary data for each pixel stored in this way is provided to neighboring pixels.

[0061] Binary data for each pixel stored in the flip-flop unit (122) can be transmitted to the expansion means (123) and the contraction means (124), so that expansion and contraction operations can be performed for each pixel.

[0062] FIG. 5 and FIG. 6 are diagrams illustrating expansion and contraction processes of a pixel array according to one embodiment of the present invention, respectively. FIG. 5 and FIG. 6 illustrate expansion and contraction of binary data converted from first image data as examples, but are not limited thereto. For example, expansion and contraction may be performed in the same manner for binary data converted from second image data.

[0063] Referring to FIG. 2, the expansion means (123) can be implemented as an n-input NAND gate.

[0064] The expansion means (123) receives binary data of the target pixel from the flip-flop unit (122) and inputs binary data of a set number of surrounding pixels (e.g., 4) from the surrounding pixels.

[0065] According to one embodiment, the expansion means (123) performs expansion to convert target pixels in a white area of ​​an eye image into black pixels (0) based on binary data of input surrounding pixels.

[0066] Referring to FIG. 5, the expansion means (123) performs expansion to convert the target pixel (510, 520) in the white area into a black pixel (0) if at least one of a predetermined number of surrounding pixels (e.g., 4) for the target pixel in the white area is a black pixel (0) in the binary data per pixel of the captured image data (eye image). In other words, the expansion means (123) can operate as an All-1 detector that maintains a white pixel (1) only when all binary data of a predetermined number of surrounding pixels based on the target pixel (510, 520) in the white area are white pixels (1).

[0067] For example, in the case of Example 1 of FIG. 5, since there is at least one black pixel (0) among the four pixels (511, 512, 513, 514) surrounding the target pixel (510) in the white area, expansion is performed by converting the target pixel (510) in the white area into a black pixel (0) in the expansion means (123). In this way, the size of the black area can be expanded through expansion.

[0068] In contrast, in the case of Example 2 of Fig. 5, since the four pixels (521, 522, 523, 524) surrounding the target pixel (520) in the white area are all white pixels (1), they are detected (All-1 detector) and the target pixel (520) in the white area is maintained as is.

[0069] However, this is an example and is not limited thereto, and as another example, the expansion means (123) may be implemented so that expansion is performed when a predetermined number or more of surrounding pixels (e.g., 2 or more, which is half of the 4 surrounding pixels) are black pixels (0).

[0070] Referring to FIG. 2, the contraction means (124) can be implemented as an n-input NOR gate.

[0071] The contraction means (124) receives binary data of a target pixel from the flip-flop unit (122) and inputs binary data of a set number of surrounding pixels (e.g., 4) from the surrounding pixels.

[0072] According to one embodiment, the shrinking means (124) can perform shrinking to convert a target pixel in a black area of ​​an eye image into a white pixel (1) based on binary data of input surrounding pixels.

[0073] Referring to FIG. 6, the shrinking means (124) performs shrinking to convert the target pixel (610, 620) in the black area into a white pixel (1) when at least one of a predetermined number (e.g., 4) of surrounding pixels with respect to the target pixel (610, 620) in the black area is a white pixel (1). In other words, the shrinking means (124) can operate as an All-0 detector that maintains a black pixel (0) only when the binary data of a predetermined number of surrounding pixels based on the target pixel (610, 620) in the black area are all black pixels (0).

[0074] For example, in the case of Example 3 of FIG. 6, since there is at least one white pixel (1) among the four pixels (611, 612, 613, 614) surrounding the target pixel (610) in the black area, expansion is performed by the shrinking means (124) to convert the target pixel (610) in the black area into a white pixel (1). In this way, the size of the black area can be reduced by shrinking.

[0075] In contrast, in the case of Example 4 of Fig. 6, since the four pixels (621, 622, 623, 624) surrounding the target pixel (620) in the black area are all black pixels (0), this is detected (All-0 detector) and the target pixel (620) in the black area is maintained as is.

[0076] However, this is an example and is not limited thereto, and as another example, the shrinking means (124) may be implemented so that shrinking is performed when a predetermined number or more of surrounding pixels (e.g., 2 or more, which is half of the 4 surrounding pixels) are white pixels (1).

[0077] The second multiplexer (125) selects a result value of either the expansion means (123) or the contraction means (124) using a control signal (CTRL) and transmits the result value to the first multiplexer (121). For example, according to a predetermined expansion and contraction algorithm, the result value of the expansion means (123) may be selected in the expansion step, and the result value of the contraction means (124) may be selected in the contraction step.

[0078] The first multiplexer (121) uses a selection signal (sel) to transmit binary data for each pixel, which has been changed according to expansion or contraction transmitted from the second multiplexer (125), to the flip-flop unit (122).

[0079] The flip-flop unit (122) stores, i.e., updates, the binary data output from the first multiplexer (121) according to the clock (CLK) operation. Accordingly, binary data (new result values) that have been changed (updated) according to expansion or contraction for each pixel can be stored in the flip-flop unit (122). The binary data for each pixel stored in this way can be provided to the surrounding pixels.

[0080] According to one embodiment, binary data for each pixel based on the output value of the processing unit (120) according to expansion and contraction is summed for each pixel column and pixel row, and contraction and expansion are terminated according to the sum result. Then, when expansion and contraction are terminated, the pupil position or reflection position is determined corresponding to the position of the black pixel (0) according to the final output value of the pixel array (100).

[0081] FIGS. 7 and 8 are diagrams relating to summation of binary data and determination of an end point in a pixel array according to an embodiment of the present invention. FIG. 9 is a diagram relating to determination of a pupil position or a reflection position in a pixel array according to an embodiment of the present invention. FIGS. 8 and 9 illustrate an example in which expansion and contraction of binary data converted from first image data are terminated and a pupil position is determined, but the present invention is not limited thereto. For example, expansion and contraction may be terminated and a reflection position may be determined in the same manner in the case of binary data converted from second image data.

[0082] Referring to FIG. 7, capacitors (C1, C2, C3, C4, C5) are provided in each column and row of the pixel array (100), so that a representative value of binary data can be stored for each column or row.

[0083] In each capacitor (C1, C2, C3, C4, C5), 1 is stored if at least one of the pixel values ​​of the pixels in the corresponding column or row, that is, binary data, is a black pixel (0). In other words, in each capacitor (C1, C2, C3, C4, C5), 0 is stored only if all of the pixel values ​​of the pixels in the corresponding column or row, that is, binary data, are white pixels (1). For example, if there is at least one black pixel (0) among the binary data of the pixels constituting the i-th row (101) of the pixel array (100), 1 is stored in the capacitor C1. As another example, if all of the binary data of the pixels constituting the j-th row (102) of the pixel array (100) are white pixels (1), 0 may be stored in the capacitor C4.

[0084] Referring to FIG. 1, a gaze tracking sensor (1) according to one embodiment includes a column register (210) in which representative values ​​of binary data of each row are stored in a column format, and a row register (220) in which representative values ​​of binary data of each column are stored in a row format. Here, capacitors C1, C2, and C5 may constitute a column register (210), and capacitors C3, C4, and C5 may constitute a row register (220).

[0085] Referring to (a) of FIG. 8, it can be confirmed that the column register (210) stores 0,1,1,1,1,1,1,0 corresponding to the representative values ​​of capacitors C1, C2, 쪋, and C5, and the row register (220) stores 0,1,1,1,1,1,1,0 corresponding to the representative values ​​of capacitors C3, C4, 쪋, and C5.

[0086] The gaze tracking sensor (1) includes a termination decision unit (300).

[0087] The termination decision unit (300) includes a column adder (310), a row adder (320), a matrix adder (full adder) (330), and a comparator (340).

[0088] The column summing unit (310) counts, i.e., adds, black pixels (0) among the representative values ​​of the column register (210) to calculate the column sum value. The row summing unit (320) counts, i.e., adds, black pixels (0) among the representative values ​​of the row register (220) to calculate the row sum value. The matrix summing unit (330) adds the column sum value and the row sum value to calculate the overall sum value.

[0089] The comparison unit (340) compares the total sum value produced by the matrix summation unit (330) with a predetermined reference value (e.g., 5), and the end of expansion and contraction can be determined based on the comparison result (output value) of the comparison unit (340).

[0090] According to one embodiment, the point in time when the total sum becomes smaller than the reference value as a result of the comparison by the comparison unit (340) is determined as the end point of the expansion and contraction operation.

[0091] For example, referring to (a) of FIG. 8, since six representative values ​​having a value of 1 are stored in the column register (210), 6 column sum values ​​are calculated in the column summing unit (310). Since six representative values ​​having a value of 1 are stored in the row register (220), 6 row sum values ​​are calculated in the row summing unit (320). In the matrix summing unit (330), the column sum values ​​and the row sum values ​​are added to calculate the overall sum value of 12ea. The comparison unit (340) compares the overall sum value of 12ea with a predetermined reference value (e.g., 5ea) and outputs a signal corresponding to the comparison result. For example, when the overall sum value is greater than or equal to the reference value (12ea > 5ea), the comparison unit (340) outputs high (1), and in response to this, the pixel array (100) can perform the contraction and expansion operation again (go on).

[0092] Accordingly, as expansion and contraction are continuously performed, the binary data of each pixel of the pixel array (100) can be converted as shown in (b) of FIG. 8. For example, since two representative values ​​having a value of 1 are stored in the column register (210), 2ea of ​​column sum values ​​are calculated in the column sum unit (310). Since two representative values ​​having a value of 1 are stored in the row register (220), 2ea of ​​row sum values ​​are calculated in the row sum unit (320). In the matrix sum unit (330), the column sum value and the row sum value are added to calculate the overall sum value of 2ea. The comparison unit (340) compares the overall sum value 4ea with a predetermined reference value (e.g., 5ea) and outputs a signal corresponding to the comparison result. For example, if the total sum is less than the reference value (4ea < 5ea), the comparison unit (340) outputs a low (0), and in response to this, the contraction and expansion operation of the pixel array (100) can be terminated (stop).

[0093] In the gaze tracking sensor (1) according to one embodiment of the present invention, the termination of the expansion and contraction operation for the pixel array (100) is determined by the termination decision unit (300) as described above. Here, the total sum value (e.g., 4ea) calculated at the time when the contraction and expansion operation is terminated corresponds to the number of black pixels (0) of the pixel array (100) as the final output value. That is, the termination decision unit (300) can terminate the expansion and contraction operation when the number of black pixels (0) becomes smaller than a predetermined reference number (e.g., 5ea) as expansion and contraction are performed in the pixel array (100).

[0094] Referring to FIG. 1, a gaze tracking sensor (1) according to one embodiment includes a column selection unit (230) that selects a column position of a black pixel (0) in a pixel array (100), and a row selection unit (240) that selects a row position of a black pixel (0).

[0095] Referring to Fig. 9, when expansion and contraction are completed, the column selection unit (230) obtains column position (710) information of a black pixel (0) from the values ​​(e.g., 0,0,0,1,1,0,0,0) stored in the column register (210). In addition, the row selection unit (240) obtains row position (720) information of a black pixel (0) from the values ​​(e.g., 0,0,0,1,1,0,0,0) stored in the row register (220). In addition, the column and row position information (700) thus obtained can be output (provided) as a pupil position or a reflection position. Here, the pupil position corresponds to the pupil center, and the reflection position corresponds to the glint center, respectively.

[0096] If there is one black pixel (0) among the values ​​of the column register (210), the column position of the corresponding pixel is selected by the column selection unit (230). Similarly, if there is one black pixel (0) among the values ​​of the row register (220), the row position of the corresponding pixel is selected by the row selection unit (240).

[0097] According to one embodiment, based on the final output value of the processing unit (120), the column selection unit (230) can select the column position of one of the plurality of black pixels, and the row selection unit (240) can select the row position of one of the plurality of black pixels.

[0098] Specifically, when there are multiple black pixels (0) among the values ​​of the column register (210), the position of one black pixel (0) (e.g., j=4) can be selected by the column selection unit (230). In addition, when there are multiple black pixels (0) among the values ​​of the row register (220), the position of one black pixel (0) (e.g., i=4) can be selected by the row selection unit (240). Accordingly, as illustrated in FIG. 9, the positions of the selected columns and rows (e.g., j=4, i=4) can be output as information on the pupil position or the reflection position.

[0099] In one embodiment, when the gaze tracking sensor (1) is in pupil position tracking mode, the pupil position is determined based on the position information of the output columns and rows. In one embodiment, when the gaze tracking sensor (1) is in reflection position tracking mode, the reflection position is determined based on the position information of the output columns and rows.

[0100] Fig. 10 is a flowchart of an operation for tracking a pupil position or a reflection position in a gaze tracking sensor according to one embodiment of the present invention. Figs. 11 and 12 are drawings each illustrating an example of tracking a pupil position and a reflection position in an eye image according to one embodiment of the present invention.

[0101] Referring to FIG. 10, in operation 810, the gaze tracking sensor (1) captures image data of an eye image pixel by pixel. Here, the captured image data includes first image data and second image data. That is, the image data captured in operation 810 is either the first image data or the second image data.

[0102] For example, referring to FIG. 11, in operation 810, the gaze tracking sensor (1) can capture first image data (910) by capturing an image of the user's eyes by a capture unit in pupil position tracking mode. As another example, referring to FIG. 12, in operation 810, the gaze tracking sensor (1) can capture second image data (1010) by irradiating external light (e.g., infrared) to the user's eyes in reflection position tracking mode.

[0103] In operation 820, the gaze tracking sensor (1) acquires binary data of the eye image.

[0104] The gaze tracking sensor (1) can obtain binary data through a binarization unit (110) provided in each pixel of the pixel array (100).

[0105] For example, the first image data (910) captured in operation 810 can be converted into binary data (920) by sequentially passing through the comparator (111) and inverter (112) of the binarization unit (110) in operation 820. As another example, the second image data (1010) captured in operation 810 can be converted into binary data (1020) by sequentially passing through the comparator (111) and buffer gate (113) of the binarization unit (110) in operation 820.

[0106] Referring to FIGS. 11 and 12, it can be confirmed that the binary data (920) of the first image data obtained in operation 820 and the binary data (1020) of the second image data are inverted from each other. That is, as illustrated in FIG. 11, the binary data (920) of the first image data (910) expresses a dark part such as a pupil as a black pixel (0), whereas as illustrated in FIG. 12, the binary data (1020) of the second image data (1010) expresses a bright part where external light is reflected (glint) as a black pixel (0).

[0107] In operation 830, the pixel array (100) performs expansion and contraction on the binary data acquired in operation 820 for each pixel based on the binary data of the surrounding pixels. Here, the binary data (920) of the first image data (910) and the binary data (1020) of the second image data (1010) are inverted, so that both the pupil position and the reflection position can be tracked through the expansion and contraction process of the same principle.

[0108] Binary data (920, 1020) for each pixel obtained in operation 820 is transmitted to the flip-flop unit (122) through the first multiplexer (210), and is transmitted from the flip-flop unit (122) to adjacent pixels and the expansion means (123) and contraction means (124), so that expansion and contraction for each pixel can be performed in operation 830.

[0109] According to one embodiment, in operation 830, when at least one of a predetermined number of surrounding pixels (e.g., 4) for a target pixel in a white area of ​​binary data (920, 1020) is a black pixel (0), expansion is performed by the expansion means (123) to convert the target pixel in a white area into a black pixel (0). In addition, in operation 830, when at least one of a predetermined number of surrounding pixels (e.g., 4) for a target pixel in a black area of ​​binary data (920, 1020) is a white pixel (1), contraction is performed to convert the target pixel in a black area into a white pixel (1).

[0110] According to one embodiment, when the gaze tracking sensor (1) is in pupil position tracking mode, expansion and contraction may be performed in the order of a first contraction step, an expansion step, and a second contraction step according to a predetermined algorithm for each pixel of the binary data (920) in operation 830. Referring to FIG. 11, in the pupil position tracking mode, as these multiple contraction steps are repeated, it can be confirmed that the binary data (930, 940, 950) gradually changes so that the black area at the edge changes to a white pixel (1) and the black area gradually decreases toward the center estimated as the pupil position.

[0111] According to one embodiment, when the gaze tracking sensor (1) is in the reflection position tracking mode, in operation 830, an expansion step may be first performed and then a contraction step may be performed for each pixel of the binary data (1020) according to a predetermined algorithm. Referring to FIG. 12, in the reflection position tracking mode, it can be confirmed that through this expansion step, the surrounding white area where external light is reflected is converted into a black pixel (0), and then the binary data (1030, 1040) gradually changes so that the black area is contracted toward a point estimated as the reflection position.

[0112] In operation 840, the gaze tracking sensor (1) adds the binary data that was expanded and contracted in operation 830 by column and row.

[0113] According to one embodiment, in operation 840, a column summation unit (310) counts (sums) black pixels (0) in a column register (210) in which representative values ​​of binary data are stored in a column format for each row of a pixel array (100), thereby obtaining a column summation value, and a row summation unit (320) counts (sums) black pixels (0) in a row register (220) in which representative values ​​of binary data are stored in a row format for each column of a pixel array (100), thereby obtaining a row summation value. Then, in operation 840, a column summation value and a row summation value are summed by a matrix summation unit (330), thereby deriving an overall summation value, and the overall summation value thus derived can be utilized to determine the end point of the expansion and contraction operations of operation 830.

[0114] In operation 850, the gaze tracking sensor (1) compares the number of black pixels (0) with a predetermined reference number (e.g., 5ea) by referring to the total sum derived in operation 840.

[0115] In operation 850, if the total sum is greater than or equal to the reference number, i.e., the reference value of the comparison unit (340), the pixel array (100) performs the expansion and contraction operations of operation 830 again.

[0116] When the total sum, i.e. the number of black pixels (0) in operation 850, becomes smaller than the reference number, i.e. the reference value of the comparison unit (30), the pixel array (100) terminates expansion and contraction.

[0117] And, based on the final output value when expansion and contraction are completed in operation 860, the location information of the black pixel (0) is output.

[0118] According to one embodiment, at the point where expansion and contraction are terminated, the column position (e.g., j=4) of the black pixel (0) may be selected by the column selection unit (230), and the row position (e.g., i=4) of the black pixel (0) may be selected by the row selection unit (240). Accordingly, in the binary data (950, 1040) corresponding to the final output value at the point where expansion and contraction are terminated, the positions of the selected column and row (e.g., j=4, i=4) may be output (provided) as information on the pupil position or the reflection position.

[0119] In the eye tracking sensor (1) according to one embodiment of the present invention as described above, both pupil position and reflection position can be tracked using mutually inverted binary data, so that accurate gaze information including the user's gaze direction can be quickly and easily obtained. Here, by performing expansion and contraction on a pixel-by-pixel basis, peripheral influences or noise are removed from the eye image, and accurate gaze tracking of the center of the pupil or reflection position is facilitated.

[0120] In addition, by implementing a gaze tracking sensor (1) capable of tracking both pupil position and reflection position in the form of a single chip, the number of parts can be minimized and the product can be miniaturized accordingly, thereby reducing manufacturing costs.

[0121] In addition, according to the above embodiment, by implementing all circuit configurations, such as the processing unit (120) constituting the pixel array (100) and the termination decision unit (300) operating based on the output value of the processing unit (120), as digital circuits, the eye tracking sensor (1) operates stably while the chip can be more easily miniaturized.

[0122] Meanwhile, according to one embodiment of the present invention, a separate capacitor may be further provided in the eye tracking sensor (1) for storing the original pixel value when image data is captured before expansion or contraction is performed for each pixel of the pixel array (100).

[0123] Since the original pixel values ​​of the initially captured image data are continuously stored in these separate capacitors, even if the pixel values ​​of specific pixels change due to subsequent shrinkage or expansion, the eye tracking sensor (1) can perform a function or operation utilizing the corresponding information (original pixel values) when necessary.

[0124] Here, as another embodiment, each pixel of the pixel array may be configured as an analog circuit to store the original pixel value and the changed pixel value, respectively.

[0125] Fig. 13 illustrates a circuit diagram of a pixel array according to another embodiment of the present invention.

[0126] Referring to FIG. 13, a pixel array (100a) according to another embodiment may include a pixel value detection unit (110a) that detects a pixel value of an eye image captured for each pixel, and a pixel value processing unit (120a) that receives and processes a value from the pixel value detection unit (110a).

[0127] The pixel value detection unit (110a) can detect and store, for example, a value corresponding to the output voltage of a photodiode (PD) that emits light according to the brightness, i.e., the intensity, of each pixel of the captured eye image as a raw pixel value.

[0128] According to one embodiment, the pixel value detection unit (110a) can detect the pixel values ​​of the original pixels of the eye image when the image is captured before expansion or contraction is performed for each pixel, and store and provide the detected original pixel values ​​to the pixel value processing unit (120a). The pixel value processing unit (120a) can store the original pixel values ​​received from the pixel value detection unit (110a).

[0129] Thereafter, in the contraction or expansion phase, the pixel value processing unit (120a) may receive a signal to change the pixel value from the surrounding adjacent pixels. In this case, the pixel value stored in the pixel value processing unit (120a) in the initial capture phase may be changed, i.e. updated, so that the pixel value detection unit (110a) and the pixel value processing unit (120a) may store different values.

[0130] Here, even if the pixel value of a specific pixel changes due to contraction or expansion, the pixel value detection unit (110a) continues to store the original pixel value of the initially captured image. By separately storing the information of the original pixels of the captured eye image in this way, even if the value of a specific pixel changes due to contraction or expansion in the future, the eye tracking sensor (1) can perform a function or operation utilizing the information when necessary.

[0131] In addition, as shown in Fig. 13, the pixel array (100a) has a pixel value detection unit (110a) and a pixel value processing unit (120a) that are each independently powered by a power source (V DD ) can be configured to supply the overall power load.

[0132] Meanwhile, according to another embodiment of the present invention, the gaze tracking sensor (1) can be implemented as a multi-layer circuit chip module.

[0133] FIG. 14 and FIG. 15 are drawings showing the configuration of a circuit chip module of a gaze tracking sensor according to another embodiment of the present invention.

[0134] Referring to FIGS. 14 and 15, in a pixel array (100a) according to another embodiment, the pixel value detection unit (110a) and the pixel value processing unit (120a) may be configured in different layers (or stacks).

[0135] For example, a pixel value detection unit (110a) that performs image detection may be provided on an upper layer (upper plate), and a pixel value processing unit (120a) that outputs values ​​(e.g., changed pixel values) related to pupil position or reflection position tracking may be provided on a lower layer (lower plate). The respective circuits provided on the upper and lower plates may be connected to each other through vias or the like.

[0136] As another example, in the eye tracking sensor (1) according to the above-described embodiment, the component for capturing image data and the component for processing the captured image data may be respectively configured in different layers. That is, the capturing unit for capturing image data may be provided in the upper layer (upper plate), and the binarization unit (110) and the processing unit (120) for processing the image data to output information on the pupil position or reflection position may be respectively provided in the lower layer (lower plate). The respective circuits provided in the upper plate and the lower plate may be connected to each other through vias or the like.

[0137] By separating the circuit configuration into separate layers in this way, the size of the chip can be further reduced due to increased circuit density.

[0138] Although the present invention has been described in detail through preferred embodiments, the present invention is not limited thereto and may be implemented in various ways within the scope of the claims.

Claims

1. In the gaze tracking sensor, It includes a pixel array composed of an MХN cell neural network having M pixel columns and N pixel rows, The above pixel array is, A capture unit that captures image data of an eye image pixel by pixel; and An eye tracking sensor including a processing unit that obtains binary data for each pixel of the captured image data, and performs expansion to convert a white area of ​​the eye image into a black pixel and contraction to convert a black area of ​​the eye image into a white pixel based on binary data of a predetermined number of surrounding pixels for the binary data for each pixel, thereby providing information on a reflection location due to external light in the eye image.

2. In paragraph 1, The above pixel array includes an inverter that selectively inverts binary data for each pixel, The above processing unit performs the expansion and contraction on binary data for each pixel inverted through the inverter, thereby providing information on the pupil position in the eye image.

3. In paragraph 1 or 2, The above processing unit, For a target pixel in the white area, if at least one of the surrounding pixels is black, an expansion means for performing the expansion to convert the target pixel to black; and A gaze tracking sensor including a shrinking means for performing the shrinking to convert the target pixel in the black area to white when at least one of the surrounding pixels is white.

4. In paragraph 1 or 2, A gaze tracking sensor further comprising a termination decision unit that sums binary data for each pixel based on the output value of the processing unit according to the expansion and contraction for the entire pixel column and pixel row, and terminates the expansion and contraction when the number of black pixels according to the sum becomes smaller than a predetermined reference number.

5. In paragraph 4, A gaze tracking sensor further comprising a column selection unit that selects a column position of one of a plurality of black pixels, and a row selection unit that selects a row position of one of a plurality of black pixels, based on the final output value of the processing unit when the above expansion and contraction are terminated.

6. In a gaze tracking method using an gaze tracking sensor including a pixel array composed of an M×N cell neural network having M pixel columns and N pixel rows, An action of capturing image data of an eye image for each pixel above; An operation of obtaining binary data for each pixel of the captured image data; An operation of performing expansion to convert a white area of ​​the eye image into a black pixel and contraction to convert a black area of ​​the eye image into a white pixel based on binary data of a predetermined number of surrounding pixels for each binary data of the above pixel; and A gaze tracking method including an operation of providing information on the location of reflection by external light in the above eye image.

Images