Eye movement tracking sensor

Abstract

An eye movement tracking sensor of the present invention includes a pixel array formed as an MXN cellular neural network having M pixel columns and N pixel rows, the pixel array including a capture section for capturing image data of an eye image pixel by pixel, and a processor for obtaining binary data for each pixel of the captured image data, and performing an expansion of converting a white region in the eye image into a black pixel and a contraction of converting a black region in the eye image into a white pixel based on the binary data of a predetermined number of adjacent pixels per pixel, thereby providing information of a flash position caused by external light in the eye image.

Description

TECHNICAL FIELD

[0001] The present application relates to an eye tracking sensor, and more particularly, to an eye tracking sensor implemented as a single chip and tracking a user's gaze by finding a pupil position and a glint position caused by external light in an eye image through image processing at a pixel level. BACKGROUND

[0002] Eye tracking, i.e., tracking a position or movement of a user's gaze, is widely used in human-computer interaction (HCI) such as virtual reality (VR), and its use in various fields including marketing, medicine, and rehabilitation is gradually increasing.

[0003] In such eye tracking, an eye image is captured by an image sensor, and a method of performing image processing on the captured eye image by using a separate digital signal processing (DSP) chip or a microprocessor embedding an image processing algorithm is widely used.

[0004] However, the DSP chip requires a large amount of power consumption, and due to the performance limitation of the DSP chip for image processing, there has been a case where a large amount of time is required to detect accurate gaze information such as a pupil position.

[0005] Meanwhile, in order to accurately recognize a gaze direction of a user as gaze information, there is a case where, in addition to a pupil position in an eye image, a glint position obtained by applying infrared light to an eye and reflected by a retina is further detected, and a method of detecting the glint position by using a separate component such as an IR position sensitive detector (PSD) is sometimes used.

[0006] However, in detecting the glint position by such a separate component, as the number of components increases, it is difficult to reduce the size of the product, and there is a problem that the component cost and the printed circuit board (PCB) manufacturing cost increase. SUMMARY

[0007] TECHNICAL PROBLEM

[0008] The present application is accomplished to solve the above problems, and provides an eye tracking sensor capable of detecting both a pupil position and a glint position through a single chip.

[0009] TECHNICAL SOLUTION

[0010] An eye tracking sensor according to one embodiment includes a pixel array formed as an M x N cellular neural network having M columns of pixels and N rows of pixels, the pixel array including a capturing section to capture image data of an eye image on a pixel-by-pixel basis, and a processor to obtain binary data of each pixel of the captured image data, and to perform dilation to convert white regions in the eye image to black pixels and erosion to convert black regions in the eye image to white pixels based on binary data of a predetermined number of neighboring pixels of each pixel, thereby providing information of a glint position in the eye image.

[0011] The pixel array can include an inverter to selectively invert the binary data of each pixel, and the processor can perform the dilation and the erosion on the binary data of each pixel inverted by the inverter to provide information of a pupil position in the eye image.

[0012] The processor can include a dilation section to perform the dilation to convert a target pixel of a white region to a black pixel when at least one of the neighboring pixels is black, and an erosion section to perform the erosion to convert a target pixel of a black region to a white pixel when at least one of the neighboring pixels is white.

[0013] A termination determination section can be further included to sum the binary data of each pixel of all the columns of pixels and all the rows of pixels based on output values of the processor according to the dilation and the erosion, and to terminate the dilation and the erosion when a number of black pixels according to the summation becomes less than a predetermined reference number.

[0014] When the dilation and the erosion are terminated, a column selection section can be further included to select a column position of any one of a plurality of black pixels based on final output values of the processor, and a row selection section to select a row position of any one of the plurality of black pixels based on the final output values of the processor.

[0015] Meanwhile, an eye tracking method using an eye tracking sensor including a pixel array formed as an M x N cellular neural network having M columns of pixels and N rows of pixels includes capturing image data of an eye image on a pixel-by-pixel basis, obtaining binary data of each pixel of the captured image data, performing dilation to convert white regions in the eye image to black pixels and erosion to convert black regions in the eye image to white pixels based on binary data of a predetermined number of neighboring pixels of each pixel, and providing information of a glint position in the eye image.

[0016] Advantageous Effects

[0017] According to such an embodiment of the present application, by providing an eye tracking sensor capable of detecting both a pupil position and a glint position, more accurate gaze information, including a gaze direction of a user, can be obtained more quickly.

[0018] Furthermore, by implementing the eye-tracking sensor as a single chip, the number of components can be minimized, thereby facilitating the reduction of product size and manufacturing costs.

[0019] Furthermore, by implementing each configuration of the eye-tracking sensor as a digital circuit, power consumption can be reduced and chip size can be reduced more easily. Attached Figure Description

[0020] Figure 1 This is a block diagram showing the configuration of an eye-tracking sensor according to an embodiment of the present invention.

[0021] Figure 2 A circuit diagram of a pixel array according to an embodiment of the present invention is shown.

[0022] Figure 3 and Figure 4 This diagram illustrates the acquisition of binary data of an eye image, which detects the pupil position and flash position in a pixel array, according to an embodiment of the present invention.

[0023] Figure 5 and Figure 6 These are diagrams showing the expansion and contraction processes of the pixel array according to embodiments of the present invention.

[0024] Figure 7 and Figure 8 This is a diagram relating to the summation of binary data in a pixel array according to an embodiment of the present invention and the determination of the termination point.

[0025] Figure 9 This diagram relates to the determination of the pupil position or flash position in a pixel array according to an embodiment of the present invention.

[0026] Figure 10 This is a flowchart illustrating the tracking of pupil position or flash position in an eye-tracking sensor according to an embodiment of the present invention.

[0027] Figure 11 and Figure 12 These are diagrams showing examples of tracking the pupil position and flash position in an eye image according to an embodiment of the present invention.

[0028] Figure 13 A circuit diagram of a pixel array according to another embodiment of the present invention is shown.

[0029] Figure 14 and Figure 15 This illustrates the configuration of a circuit chip module for an eye-tracking sensor according to another embodiment of the present invention. Detailed Implementation

[0030] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art to which this invention pertains can easily implement them.

[0031] Figure 1 This is a block diagram showing the configuration of an eye-tracking sensor according to an embodiment of the present invention. Figure 2 A circuit diagram of a pixel array according to an embodiment of the present invention is shown.

[0032] Eye-tracking sensors (1) are placed in electronic devices such as head-mounted display devices (HMDs) or smart glasses and provide information about the user's gaze.

[0033] According to one embodiment, the eye-tracking sensor (1) is implemented as a single-chip complementary metal oxide silicon (CMOS) image sensor (CIS). A CMOS image sensor is an image sensor manufactured using CMOS fabrication technology, which enables the integration of circuitry for image signal processing onto a single chip, facilitating module miniaturization through a System on Chip (SoC). Furthermore, the CMOS image sensor is capable of pixel-level image processing via a pixel array (100) described later.

[0034] According to an embodiment of the present invention, the eye-tracking sensor (1) is capable of performing two functions. First, the eye-tracking sensor (1) provides the user's gaze information based on the captured image data, i.e., based on the eye image. Second, the eye-tracking sensor (1) outputs the captured image data through an image output unit (400). In one embodiment, the image output unit (400) includes a Correlated Double Sampling (CDS) circuit and an A / D converter (ADC), and is capable of converting the captured image data into a digital signal for output.

[0035] According to one embodiment, the eye-tracking sensor (1) can operate in two modes: a pupil position tracking mode for tracking the position of the pupil (eyeball) in an eye image, and a flash position tracking mode for tracking the flash position of external light such as infrared light. That is, by tracking the flash position, the eye-tracking sensor (1) installed in the electronic device can more accurately and quickly identify the user's gaze direction.

[0036] The eye-tracking sensor (1) operates selectively in one of a pupil position tracking mode and a flash position tracking mode, and provides information on the pupil position or flash position corresponding to each operating mode. That is, according to one embodiment of the invention, the eye-tracking sensor (1) is implemented to detect (track) both the pupil position and the flash position in the image data.

[0037] The eye-tracking sensor (1) includes a pixel array (100).

[0038] Reference Figure 1 The pixel array (100) is formed as an M×N cellular neural network with M pixel columns (hereinafter also referred to as columns) and N pixel rows (hereinafter also referred to as rows). The output of each pixel in the pixel array (100) represents the value of its state, for example, a black pixel or a white pixel, and the output of each pixel can be summed by all columns and rows.

[0039] The pixel array (100) includes a capture unit (not shown) for capturing image data of an eye image pixel by pixel by imaging the user's eye. The capture unit can be implemented as a capture component in the form of an array, such that image data of each pixel can be captured by each capture unit of the capture component.

[0040] According to one embodiment, the capture unit can capture image data for each pixel in a global shutter mode. However, this is not limiting, and in another example, the capture unit can be used in a rolling shutter mode.

[0041] According to one embodiment, the capturing unit can capture image data when external light (e.g., infrared light) is applied to the user's eye. Figure 4 (Second image data). When external light is applied to the user's eye, a bright-eye phenomenon occurs, where the pupil brightens due to light reflected from the retina, and the location of external light reflected from the cornea (glint) is detected. By using the positional relationship between these two parts, the direction of eye gaze can be tracked. For example, when infrared light is applied and image data for each pixel is captured, the bright-eye phenomenon and the location of the glint are detected. The relative positional relationship between the center point of change of the bright pupil (pupil position) and the location of the glint as the eye moves up, down, left, or right can be used to track the direction of gaze.

[0042] According to one embodiment, the pixel array (100) provides information on the pupil position or flash position in an eye image, i.e., in the captured image data. For example, the pixel array (100) is capable of outputting information on the pupil position or flash position corresponding to a pupil position tracking mode or a flash position tracking mode.

[0043] Reference Figure 2 The pixel array (100) includes a binarization unit (110) for each pixel and a processor (120).

[0044] According to one embodiment, a binarization unit (110) and a processor (120) are connected to each capture unit, enabling image data for each pixel to be processed.

[0045] Figure 3 and Figure 4 This diagram illustrates the acquisition of binary data of an eye image, which detects the pupil position and flash position in a pixel array, respectively, according to an embodiment of the present invention.

[0046] Reference Figure 2 The binarization unit (110) includes a comparator (111), which compares the pixel data of each pixel of the captured image data with a threshold and outputs binary data.

[0047] According to one embodiment or another example, the binarization unit (110) includes an inverter (112), i.e., a NOT gate, to invert the binary data output from the comparator (111). Here, the inverter (112) is capable of selectively inverting the binary data output from the comparator (111). Furthermore, the binarization unit (110) may include a buffer gate (113) that outputs the binary data output from the comparator (111) as is.

[0048] According to one embodiment, binary data output from comparator (111) is input to one of inverter (112) and buffer gate (113). For example, in pupil position tracking mode, binary data can be input to inverter (112), and in flash position tracking mode, binary data can be input to buffer gate (113). Here, the binary data receives an external signal indicating either pupil position tracking mode or flash position tracking mode, and can be input to one of inverter (112) and buffer gate (113) accordingly. However, this is merely an example, and binary data can be input to buffer gate (113) in pupil position tracking mode and to inverter (112) in flash position tracking mode. Furthermore, in another embodiment of the pixel array (100), each pixel can be implemented in such a way that a bypass path to inverter (112) is provided without a buffer gate.

[0049] The following describes the cases where binary data is input to an inverter (112) during pupil position tracking and to a buffer gate (113) during flash position tracking.

[0050] Reference Figure 3 and Figure 4 The comparator (111) compares the pixel data of each pixel of the captured image data (pixel input) with a predetermined threshold (external threshold) and outputs binary data corresponding to the comparison result, such as 1 or 0.

[0051] In one embodiment, comparator (111) outputs 0 when the pixel data is equal to or less than the threshold, and outputs 1 when the pixel data is greater than the threshold. Here, 0 is defined as the value representing a black pixel, and 1 is defined as the value representing a white pixel.

[0052] Reference Figure 3 The image data of the eye image captured in the pupil position tracking mode (hereinafter also referred to as the first image data) has pixel data of the pixels corresponding to the first region (PA1) of the pupil position, which has a relatively larger value than other pixels.

[0053] The pixel data of each pixel in the first image data is converted into binary data of 1 or 0 by a comparator (111) and inverted by an inverter (112). Therefore, as Figure 3As shown, the binary data (binarized pixel output) of the first image data can be as follows: the pixel data of the first region (PA1) with a value greater than the threshold is converted to 0 (Px.OUT=0), and the pixel data of other regions is converted to 1 (Px.OUT=1). The binarization unit (110) outputs the binary data of the first image data converted as described above by the comparator (111) and the inverter (112).

[0054] Reference Figure 4 The image data of the eye image captured in the flash position tracking mode (hereinafter also referred to as the second image data) has pixel data of the pixels corresponding to the second region (PA2) of the flash position, which has a relatively smaller value than the other pixels.

[0055] The pixel data of each pixel in the second image data is converted into binary data of 1 or 0 by a comparator (111) and then passed through a buffer gate (113). Therefore, as... Figure 4 As shown, the binary data (binarized pixel output) of the second image data can be such that the pixel data of the second region (PA2) having a value equal to or less than a threshold is converted to 0 (Px.OUT=0), and the pixel data of other regions is converted to 1 (Px.OUT=1). The binarization unit (110) outputs the binary data of the second image data converted as described above by the comparator (111) and the buffer gate (113).

[0056] According to one embodiment of the present invention, the first image data and the second image data are provided as inverted binary data by the binarization unit (110).

[0057] For example, in the first image data, pixel data of a first region (PA1) with a value greater than a threshold can be converted to black pixels (0), and other pixel data can be converted to white pixels (1). In contrast, in the second image data, pixel data of a second region (PA2) with a value equal to or less than a threshold can be converted to black pixels (0), and other pixel data can be converted to white pixels (1).

[0058] The processor (120) processes the captured image data.

[0059] According to one embodiment, the processor (120) processes the binary data of each pixel output from the binarization unit (110), and specifically, it may perform expansion to convert the white area of ​​the eye image into black pixels (0), or it may perform shrinking to convert the black area of ​​the eye image into white pixels (1).

[0060] In an eye-tracking sensor (1) according to an embodiment of the present invention, each pixel of the pixel array (100) is expanded and contracted as described above, and the pupil position and flash position can be identified through such pixel-level interaction.

[0061] Reference Figure 2 The processor (120) includes a first multiplexer (121), a trigger section (DFF) (122), an expansion section (all-1 detector) (123), a contraction section (all-0 detector) (124), and a second multiplexer (125).

[0062] The first multiplexer (121) uses a selection signal (sel) to pass the binary data of each pixel output from the binarization unit (110) to the trigger unit (122).

[0063] The flip-flop unit (122) is implemented as a D flip-flop (DFF) and stores the binary data output from the first multiplexer (121) synchronously with the clock signal (CLK). Here, the flip-flop unit (122) can store binary data for one clock cycle. The stored binary data for each pixel is provided to neighboring pixels.

[0064] The binary data of each pixel stored in the trigger section (122) is passed to the expansion section (123) and the contraction section (124) so ​​that expansion and contraction operations can be performed on each pixel.

[0065] Figure 5 and Figure 6 These are diagrams showing the expansion and contraction processes of a pixel array according to an embodiment of the present invention. Figure 5 and Figure 6 The example illustrates the expansion and contraction of binary data converted from the first image data, but this is not limiting, and, for example, the expansion and contraction of binary data converted from the second image data can also be performed in the same way.

[0066] Reference Figure 2 The extension part (123) can be implemented as an n-in NAND gate.

[0067] The extension unit (123) receives binary data of the target pixel from the trigger unit (122) and receives a predetermined number (e.g., four) of binary data of neighboring pixels from neighboring pixels.

[0068] According to one embodiment, the expansion unit (123) performs expansion based on the binary data of the input neighboring pixels to convert the target pixel in the white area into a black pixel (0).

[0069] Reference Figure 5 The extension unit (123) performs extension in the binary data of each pixel in the captured image data (eye image) to convert the target pixel (510, 520) in the white area into a black pixel (0) when at least one of the predetermined number (e.g., four) of its neighboring pixels is a black pixel (0). In other words, the extension unit (123) can operate as an all-1 detector that retains a white pixel (1) only when the binary data of the predetermined number of neighboring pixels are all white pixels (1) relative to the target pixel (510, 520) in the white area.

[0070] For example, in Figure 5 In Example 1, since at least one of the four adjacent pixels (511, 512, 513, 514) of the target pixel (510) in the white area is a black pixel (0), the expansion unit (123) performs expansion to convert the target pixel (510) in the white area into a black pixel (0). In this way, the size of the black area can be increased by expansion.

[0071] In contrast, Figure 5 In Example 2, since the four adjacent pixels (521, 522, 523, 524) of the target pixel (520) in the white area are all white pixels (1), the extension (123) detects this (all-1 detector) and keeps the target pixel (520) in the white area unchanged.

[0072] However, this is merely an example and not a limitation. In another example, the extension (123) can be implemented to perform the extension when a predetermined number or more of the adjacent pixels (e.g., two or more of four adjacent pixels) are black pixels (0).

[0073] Reference Figure 2 The contraction (124) can be implemented as an n-in NOR gate.

[0074] The contraction section (124) receives binary data of the target pixel from the trigger section (122) and receives a predetermined number (e.g., four) of binary data of adjacent pixels from adjacent pixels.

[0075] According to one embodiment, the shrinking unit (124) can perform shrinking based on the binary data of the input neighboring pixels to convert the target pixel in the black area into a white pixel (1).

[0076] Reference Figure 6 When at least one of a predetermined number (e.g., four) of the target pixel in the black area is a white pixel (1), the shrinking unit (124) performs shrinkage to convert the target pixel (610, 620) in the black area into a white pixel (1). In other words, the shrinking unit (124) can operate as an all-zero detector, maintaining the black pixel (0) only when the binary data of the predetermined number of neighboring pixels are all black pixels (0) relative to the target pixel (610, 620) in the black area.

[0077] For example, in Figure 6 In Example 3, since at least one of the four adjacent pixels (611, 612, 613, 614) of the target pixel (610) in the black area is a white pixel (1), the shrinking part (124) performs shrinkage to convert the target pixel (610) in the black area into a white pixel (1). In this way, the size of the black area can be reduced by shrinkage.

[0078] In contrast, Figure 6 In Example 4, since the four adjacent pixels (621, 622, 623, 624) of the target pixel (620) in the black area are all black pixels (0), the contraction part (124) detects this (all-zero detector) and keeps the target pixel (620) in the black area unchanged.

[0079] However, this is only an example and not a limitation. In another example, the contraction (124) can be implemented to perform contraction when a predetermined number or more of the adjacent pixels (e.g., two or more of four adjacent pixels) are white pixels (1).

[0080] The second multiplexer (125) uses a control signal (CTRL) to select one of the result values ​​of the expansion section (123) and the contraction section (124), and passes the selected result value to the first multiplexer (121). For example, according to a predetermined expansion and contraction algorithm, the result value of the expansion section (123) can be selected in the expansion stage, and the result value of the contraction section (124) can be selected in the contraction stage.

[0081] The first multiplexer (121) uses a selection signal (sel) to pass the binary data of each pixel, which changes according to the expansion or contraction transmitted from the second multiplexer (125), to the trigger section (122).

[0082] The trigger unit (122) operates according to the clock (CLK) to store, that is, to update the binary data output from the first multiplexer (121). Therefore, the trigger unit (122) can store the binary data (new result value) of each pixel that changes (updates) according to expansion or contraction. The stored binary data of each pixel can be provided to neighboring pixels.

[0083] According to one embodiment, the processor (120) sums the binary data of each pixel based on the expanded and contracted output values ​​for each pixel column and pixel row, and terminates the contraction and expansion based on the summation result. Then, when the expansion and contraction terminate, the pupil position or flash position is determined based on the position of the black pixel (0) in the final output value of the pixel array (100).

[0084] Figure 7 and Figure 8 This is a diagram related to the summation and determination of the termination point of binary data in a pixel array according to an embodiment of the present invention. Figure 9 This is a diagram related to the determination of the pupil position or flash position in a pixel array according to an embodiment of the present invention. Figure 8 and Figure 9 The example shown illustrates the case where expansion and contraction terminate with binary data converted from first image data and determine the pupil position, but this is not a limitation. For example, in the case of binary data converted from second image data, expansion and contraction can also terminate in the same way and determine the flash position.

[0085] Reference Figure 7 Capacitors (C1, C2, C3, C4, C5) are set in each column and each row of the pixel array (100), and the representative value of the binary data of each column or each row can be stored in the capacitors.

[0086] In each capacitor (C1, C2, C3, C4, C5), a 1 is stored in the binary data of the pixel corresponding to the column or row when the pixel value includes at least one black pixel (0). That is, in each capacitor (C1, C2, C3, C4, C5), a 0 is stored only when the pixel value, i.e., the binary data of the pixel corresponding to the column or row, is all white pixels (1). For example, a 1 is stored in capacitor C1 when the binary data of the pixels constituting the i-th row (101) in the pixel array (100) contains at least one black pixel (0). As another example, a 0 can be stored in capacitor C4 when the binary data of the pixels constituting the j-th row (102) in the pixel array (100) is all white pixels (1).

[0087] Reference Figure 1According to one embodiment, the eye-tracking sensor (1) includes a column register (210) in which the representative value of each row of binary data is stored in column form, and a row register (220) in which the representative value of each column of binary data is stored in row form. Here, capacitors C1, C2, and C5 constitute the column register (210), and capacitors C3, C4, and C5 may constitute the row register (220).

[0088] Reference Figure 8 As can be seen from (a), the column register (210) stores the representative values ​​0,1,1,1,1,1,1,0 corresponding to capacitors C1, C2, ..., C5, and the row register (220) stores the representative values ​​0,1,1,1,1,1,1,0 corresponding to capacitors C3, C4, ..., C5.

[0089] The eye-tracking sensor (1) includes a termination determination unit (300).

[0090] The termination determination unit (300) includes a column adder (310), a row adder (320), a full adder (330), and a comparator (340).

[0091] The column adder (310) counts, that is, sums the black pixels (0) in the representative values ​​of the column register (210) and calculates the column sum value, and the row adder (320) counts, that is, sums the black pixels (0) in the representative values ​​of the row register (220) and calculates the row sum value. The full adder (330) sums the column sum values ​​and the row sum values ​​to calculate the total sum value.

[0092] The comparator (340) compares the total sum calculated by the full adder (330) with a predetermined reference value (e.g., 5), and can terminate expansion and contraction based on the comparison result (output value) of the comparator (340).

[0093] According to one embodiment, when the comparison result of the comparator (340) indicates that the total sum becomes less than the reference value, the time point is determined as the termination point of the expansion and contraction operation.

[0094] For example, refer to Figure 8In step (a), since the six representative values ​​with a value of 1 are stored in the column register (210), the column adder (310) calculates the column sum of 6ea. Since the six representative values ​​with a value of 1 are stored in the row register (220), the row adder (320) calculates the row sum of 6ea. The full adder (330) adds the column sum and the row sum to calculate the total sum of 12ea. The comparator (340) compares the total sum of 12ea with a predetermined reference value (e.g., 5ea) and outputs a signal corresponding to the comparison result. For example, when the total sum is equal to or greater than the reference value (12ea > 5ea), the comparator (340) outputs 1, and in response, the pixel array (100) can perform shrink and expand operations again.

[0095] Therefore, as expansion and contraction continue to be performed, the binary data of each pixel in the pixel array (100) can be as follows: Figure 8 The transformation is shown in (b). For example, since two representative values ​​with a value of 1 are stored in the column register (210), the column adder (310) calculates the column sum of 2ea. Since two representative values ​​with a value of 1 are stored in the row register (220), the row adder (320) calculates the row sum of 2ea. The full adder (330) adds the column sum and the row sum to calculate the total sum of 4ea. The comparator (340) compares the total sum of 4ea with a predetermined reference value (e.g., 5ea) and outputs a signal corresponding to the comparison result. For example, when the total sum is less than the reference value (4ea < 5ea), the comparator (340) outputs 0, and in response, the shrinking and expanding operations of the pixel array (100) can be stopped.

[0096] In an eye-tracking sensor (1) according to an embodiment of the present invention, as described above, a termination determination unit (300) determines the termination of the expansion and contraction operations of the pixel array (100). Here, the total summation value (e.g., 4ea) calculated when the contraction and expansion operations are terminated corresponds to the number of black pixels (0) in the pixel array (100) as the final output value. That is, when the number of black pixels (0) in the pixel array (100) becomes less than a predetermined reference number (e.g., 5ea) as expansion and contraction are performed, the termination determination unit (300) may terminate the contraction and expansion operations.

[0097] Reference Figure 1 According to one embodiment, the eye-tracking sensor (1) includes a column selection unit (230) that selects the column position of a black pixel (0) in a pixel array (100) and a row selection unit (240) that selects the row position of a black pixel (0).

[0098] Reference Figure 9When expansion and contraction are terminated, the column selection unit (230) obtains the column position information (710) of the black pixel (0) from the value (e.g., 0,0,0,1,1,0,0,0) stored in the column register (210). Furthermore, the row selection unit (240) obtains the row position information (720) of the black pixel (0) from the value (e.g., 0,0,0,1,1,0,0,0) stored in the row register (220). The obtained column and row position information (700) can be output (provided) as the pupil position or the flash position. Here, the pupil position corresponds to the pupil center, and the flash position corresponds to the flash center.

[0099] When there is only one black pixel (0) in the column register (210), the column position of the corresponding pixel is selected by the column selection unit (230). Similarly, when there is only one black pixel (0) in the row register (220), the row position of the corresponding pixel is selected by the row selection unit (240).

[0100] According to one embodiment, based on the final output value of the processor (120), the column selection unit (230) can select the column position of any one of the plurality of black pixels, and the row selection unit (240) can select the row position of any one of the plurality of black pixels.

[0101] Specifically, when there are multiple black pixels (0) in the column register (210), a position of a black pixel (0) (e.g., j=4) can be selected by the column selection unit (230). Similarly, when there are multiple black pixels (0) in the row register (220), a position of a black pixel (0) (e.g., i=4) can be selected by the row selection unit (240). Therefore, as Figure 9 As shown, the selected column and row positions (e.g., j=4, i=4) can be output (provided) as information about the pupil position or flash position.

[0102] In one embodiment, when the eye-tracking sensor (1) is in pupil position tracking mode, the pupil position is determined based on the output column and row position information. In one embodiment, when the eye-tracking sensor (1) is in flash position tracking mode, the flash position is determined based on the output column and row position information.

[0103] Figure 10 This is a flowchart illustrating the tracking of pupil position or flash position in an eye-tracking sensor according to an embodiment of the present invention. Figure 11 and Figure 12 These are diagrams showing examples of tracking the pupil position and flash position in an eye image according to an embodiment of the present invention.

[0104] ReferenceFigure 10 In operation 810, the eye-tracking sensor (1) captures image data of the eye on a pixel-by-pixel basis. Here, the captured image data includes first image data and second image data. That is, the image data captured in operation 810 becomes either first image data or second image data.

[0105] For example, refer to Figure 11 In operation 810, the eye-tracking sensor (1) can capture first image data (910) by imaging the user's eye in pupil position tracking mode via the capture unit. As another example, see... Figure 12 In operation 810, the eye-tracking sensor (1) can capture second image data (1010) when external light (e.g., infrared light) is applied to the user's eye in flash position tracking mode.

[0106] In operation 820, the eye-tracking sensor (1) acquires binary data of the eye image.

[0107] The eye-tracking sensor (1) can acquire binary data through the binarization part (110) provided by each pixel in the pixel array (100).

[0108] For example, the first image data (910) captured in operation 810 can be converted into binary data (920) in operation 820 by a comparator (111) and an inverter (112) passing sequentially through the binarization unit (110). As another example, the second image data (1010) captured in operation 810 can be converted into binary data (1020) in operation 820 by a comparator (111) and a buffer gate (113) passing sequentially through the binarization unit (110).

[0109] Reference Figure 11 and Figure 12 It can be seen that the binary data of the first image data (920) and the binary data of the second image data (1020) acquired in operation 820 are reversed. That is, as shown... Figure 11 As shown, in the binary data (920) of the first image data (910), dark areas such as the pupil are represented as black pixels (0), however, as Figure 12 As shown, in the binary data (1020) of the second image data (1010), the bright part (glint) where external light is reflected is represented as a black pixel (0).

[0110] In operation 830, the pixel array (100) performs expansion and contraction on each pixel based on the binary data acquired in operation 820, using the binary data of adjacent pixels. Here, since the binary data (920) of the first image data (910) and the binary data (1020) of the second image data (1010) are inverted, the pupil position and flash position can both be tracked using the same principle of the expansion and contraction process.

[0111] The binary data (920, 1020) of each pixel acquired in operation 820 is passed to the trigger section (122) through the first multiplexer (210), and from the trigger section (122) to the adjacent pixels as well as the expansion section (123) and the contraction section (124), so that the expansion and contraction of each pixel can be performed in operation 830.

[0112] According to one embodiment, in operation 830, when at least one of a predetermined number (e.g., four) of the neighboring pixels of the target pixel in the white region of the binary data (920, 1020) is a black pixel (0), the expansion unit (123) performs expansion to convert the target pixel in the white region into a black pixel (0). Furthermore, in operation 830, when at least one of a predetermined number (e.g., four) of the neighboring pixels of the target pixel in the black region of the binary data (920, 1020) is a white pixel (1), the contraction unit (124) performs contraction to convert the target pixel in the black region into a white pixel (1).

[0113] According to one embodiment, when the eye-tracking sensor (1) is in pupil position tracking mode, in operation 830, each pixel of the binary data (920) is expanded and contracted in sequence according to a first contraction phase, an expansion phase, and a second contraction phase according to a predetermined algorithm. (Refer to...) Figure 13 In the pupil position tracking mode, as these multiple contraction phases are repeated, it can be seen that the binary data (930, 940, 950) gradually changes, causing the black area at the edge to be changed to white pixels (1), and the black area gradually decreases toward the center estimated as the pupil position.

[0114] According to one embodiment, when the eye-tracking sensor (1) is in flash position tracking mode, in operation 830, an expansion phase is first performed on each pixel of the binary data (1020) according to a predetermined algorithm, followed by a contraction phase. (See also...) Figure 13 In the flash position tracking mode, it can be seen that through the expansion phase, the surrounding white area where the external light is reflected is converted into black pixels (0), and then the black area gradually decreases toward the point estimated as the flash position, causing the binary data (1030, 1040) to gradually change.

[0115] In operation 840, the eye-tracking sensor (1) sums the binary data that was expanded and contracted in operation 830, column by column and row by row.

[0116] According to one embodiment, in operation 840, the column summation value is obtained by counting (summing) the black pixels (0) in the column register (210) using a column adder (310), wherein the representative value of the binary data of each row of the pixel array (100) is stored in the column register (210) in column form, and the row summation value is obtained by counting (summing) the black pixels (0) in the row register (220) using a row adder (320), wherein the representative value of the binary data of each column of the pixel array (100) is stored in the row register (220) in row form. Then, in operation 840, the total summation value is derived by summing the column summation value and the row summation value using a full adder (330), and the derived total summation value can be used to determine the termination point of the expansion and contraction operations in operation 830.

[0117] In operation 850, the eye-tracking sensor (1) compares the number of black pixels (0) with a predetermined reference number (e.g., 5) by referring to the total summation value derived in operation 840.

[0118] In operation 850, when the total summation is equal to or greater than the reference quantity, i.e. the reference value of the comparator (340), the pixel array (100) performs the expansion and contraction operation of operation 830 again.

[0119] In operation 850, when the total summation, i.e. the number of black pixels (0), becomes less than the predetermined reference number, i.e. the reference value of the comparator (340), the pixel array (100) terminates expansion and contraction.

[0120] Then, in operation 860, the position information of the black pixel (0) is output based on the final output value at the end of expansion and contraction.

[0121] According to one embodiment, at the termination of expansion and contraction, the column position (e.g., j = 4) of the black pixel (0) can be selected by the column selection unit (230), and the row position (e.g., i = 4) of the black pixel (0) can be selected by the row selection unit (240). Therefore, in the binary data (950, 1040) corresponding to the final output value at the termination of expansion and contraction, the selected column and row positions (e.g., j = 4, i = 4) can be output (provided) as the pupil position or flash position.

[0122] In an eye-tracking sensor (1) according to an embodiment of the present invention, since both the pupil position and the flash position can be tracked using binary data that is inverted from each other, accurate gaze information, including the user's gaze direction, can be obtained quickly and easily. Here, by performing expansion and contraction at the pixel level, the effects or noise around the eye image can be easily removed, and the center of the tracking aperture or the flash position can be accurately tracked.

[0123] Furthermore, by implementing the eye-tracking sensor (1) that can track the position of the pupil and the position of the flash as a single chip, the number of components can be minimized, thereby reducing the product size and manufacturing cost.

[0124] Furthermore, according to one embodiment, since all circuit configurations, such as the processor (120) constituting the pixel array (100) and the termination determination unit (300) operating based on the output value of the processor (120), are implemented as digital circuits, the eye-tracking sensor (1) can operate stably, and the miniaturization of the chip is easier to achieve.

[0125] Furthermore, according to one embodiment of the present invention, the eye-tracking sensor (1) may further include, for each pixel of the pixel array (100), a separate capacitor that stores the original pixel value captured when the image data was captured before performing expansion or contraction.

[0126] In a separate capacitor, the original pixel values ​​of the initially captured image data continue to be stored, so that even when the pixel value of a particular pixel changes due to later shrinkage or expansion, the eye-tracking sensor (1) can use the corresponding information (original pixel value) to perform functions or operations when necessary.

[0127] Here, as another embodiment, each pixel of the pixel array can be configured as an analog circuit so that the original pixel value and the changed pixel value can be stored separately.

[0128] Figure 13 A circuit diagram of a pixel array according to another embodiment of the present invention is shown.

[0129] Reference Figure 14 According to another embodiment, the pixel array (100a) may include a pixel value detection unit (110a) that detects the pixel value of the eye image captured for each pixel, and a pixel value processing unit (120a) that receives and processes the value from the pixel value detection unit (110a).

[0130] The pixel value detection unit (110a) can detect and store, for example, the value of the output voltage of a photodiode (PD) as the original pixel value, which emits light according to the brightness, i.e. the brightness of each pixel in the captured eye image.

[0131] According to one embodiment, the pixel value detection unit (110a) can detect the pixel values ​​of the original pixels of the eye image when capturing the image before performing expansion or contraction for each pixel, and can store the detected original pixel values ​​and provide them to the pixel value processing unit (120a). The pixel value processing unit (120a) can store the original pixel values ​​received from the pixel value detection unit (110a).

[0132] Subsequently, during the shrinking or expanding phase, the pixel value processing unit (120a) can receive signals from adjacent pixels that change pixel values. In this case, the pixel values ​​stored in the pixel value processing unit (120a) during the initial capture phase are changed, i.e., updated, so the pixel value detection unit (110a) and the pixel value processing unit (120a) can store different values.

[0133] Here, even when the pixel value of a particular pixel is changed due to shrinkage or expansion, the original pixel value of the initially captured image continues to be stored in the pixel value detection unit (110a). By storing the information of the original pixels of the captured eye image in this way, even when the value of a particular pixel is changed due to shrinkage or expansion, the eye-tracking sensor (1) can use the corresponding information to perform functions or operations when necessary.

[0134] In addition, such as Figure 15 As shown, the pixel array (100a) can be configured such that the pixel value detection unit (110a) and the pixel value processing unit (120a) receive power (VDD) independently, thereby reducing the overall power load.

[0135] Meanwhile, according to another embodiment of the present invention, the eye-tracking sensor (1) can be implemented as a multilayer circuit chip module.

[0136] Figure 14 and Figure 15 The configuration of a circuit chip module for an eye-tracking sensor according to another embodiment of the present invention is shown.

[0137] Reference ​ and ​ In a pixel array (100a) according to another embodiment, the pixel value detection unit (110a) and the pixel value processing unit (120a) may be disposed on different layers (or stacks).

[0138] For example, a pixel value detection unit (110a) that performs image detection can be provided on the upper layer (upper board), while a pixel value processing unit (120a) that outputs a value (e.g., a changed pixel value) related to the tracking pupil position or flash position can be provided on the lower layer (lower board). The circuits provided on the upper and lower boards can be connected to each other through vias.

[0139] As another example, in the eye-tracking sensor (1) according to the above embodiment, the configuration for capturing image data and the configuration for processing the captured image data can be arranged on different layers. That is, the capture unit for capturing image data can be arranged on the upper layer (upper plate), while the binarization unit (110) and the processor (120) for processing image data and outputting information about the pupil position or flash position can be arranged on the lower layer (lower plate). The circuits arranged on the upper plate and the lower plate can be connected to each other through vias.

[0140] By separating the circuit configuration onto different layers in this way, the chip size can be further reduced due to the increased circuit density.

Claims

1. An eye-tracking sensor, comprising: The pixel array is formed into an MXN cellular neural network with M pixel columns and N pixel rows; The pixel array includes: The capture unit is used to capture image data of the eye pixel by pixel; as well as The processor is configured to obtain binary data for each pixel of the captured image data, to perform expansion to convert white areas in the eye image into black pixels, and to perform contraction based on binary data of a predetermined number of neighboring pixels for each pixel to convert black areas in the eye image into white pixels, thereby providing information on the location of flashes caused by external light in the eye image.

2. The eye-tracking sensor of claim 1, wherein the pixel array includes an inverter for selectively inverting the binary data of each pixel, and The processor performs the expansion and contraction on the binary data of each pixel inverted by the inverter, based on the binary data of the predetermined number of neighboring pixels of each pixel, thereby providing information on the pupil position in the eye image.

3. The eye-tracking sensor according to claim 1, wherein the processor comprises: An extension unit is configured to perform the extension to convert the target pixel in the white region into a black pixel when at least one of the adjacent pixels is black; as well as A contraction section is used to perform the contraction to convert the target pixel in the black area into a white pixel when at least one of the adjacent pixels is white.

4. The eye-tracking sensor according to claim 1, further comprising a termination determination unit, configured to sum the binary data of each pixel in all pixel columns and all pixel rows based on the output value of the processor according to the expansion and the contraction, and to terminate the expansion and the contraction when the number of black pixels according to the summation becomes less than a predetermined reference number.

5. The eye-tracking sensor according to claim 4 further includes a column selection unit for selecting the column position of any one of the plurality of black pixels, and a row selection unit for selecting the row position of any one of the plurality of black pixels based on the final output value of the processor when the expansion and contraction terminate.

6. An eye-tracking method using the eye-tracking sensor, the eye-tracking sensor comprising a pixel array formed as an M×N cellular neural network having M pixel columns and N pixel rows, the method comprising: Capture image data of the eye image pixel by pixel; Obtain the binary data of each pixel in the captured image data; Based on the binary data of a predetermined number of neighboring pixels for each pixel, expansion is performed to convert white areas in the eye image into black pixels, and contraction is performed to convert black areas in the eye image into white pixels; as well as Provides information on the location of flashes caused by external light in the eye image.

Images