Apparatus and method for controlling light sources for eye tracking

Abstract

The present disclosure relates to an apparatus and a method for controlling light sources for eye tracking, wherein the method may comprise: performing user calibration by using eye tracking technology; analyzing user eye data obtained by means of the user calibration and calculating the activity of each of light sources according to the gaze of a user so as to generate an activity map for each of the light sources; checking a first gaze direction that is a gaze direction of the user identified from an image taken of the pupils of the user; checking, in the activity map for each of the light sources, control information for each of the light sources corresponding to the first gaze direction; and controlling each of the light sources on the basis of the control information.

Description

Device and method for controlling a light source for eye tracking

[0001] The following embodiments relate to techniques for eye tracking in extended reality (XR) devices.

[0002] Most extended reality (XR) devices use cameras (e.g., infrared ray (IR) cameras) and light sources (e.g., IR LEDs) to acquire eye information. This is because the gaze tracking sensor configuration can easily acquire user eye information as it is seen by a person, making analysis easy, and because IR cameras have been widely used for a long time, there are many gaze tracking studies using IR cameras. Gaze tracking technology using IR cameras and IR LEDs operates by analyzing the relationship between the light emitted by the IR LED bouncing off the user's eye and being input to the IR camera. However, since the location of the IR LED, which can form reflected light (Glint), varies depending on the position of the user's eye, the operation stability of the gaze tracking technology is usually secured by placing multiple IR LEDs around the user's eye.

[0003] Most XR devices with conventional eye tracking provide gaze tracking technology by turning on all IR LEDs or dividing them into groups of two or three and cyclically blinking them. The background of this control is to secure the largest possible glint formation area in situations where the user's eye location is unknown. However, this control method is inefficient in terms of power consumption because it uses unnecessary IR LEDs that do not form glints. In addition, it increases the difficulty of developing a reflected light detector because the light from unnecessary IR LEDs that do not form glints is applied as interference in terms of gaze tracking accuracy.

[0004] The present disclosure provides a device and method for controlling a light source for open gaze tracking.

[0005] A method for controlling a light source for gaze tracking according to one embodiment may include: performing user calibration using gaze tracking technology; analyzing user eye data acquired through the user calibration to calculate the activity of each light source according to the gaze of the user, thereby generating an activity map of each light source; confirming a first gaze direction, which is a gaze direction of the user confirmed through an image capturing the pupil of the user; confirming control information of each light source corresponding to the first gaze direction in the activity map of each light source; and controlling each light source according to the control information.

[0006] According to one embodiment, a device for controlling a light source for eye tracking may include: a camera for photographing a user's eyes; light sources for obtaining information about the user's eyes; a calibration processing unit for performing user calibration using eye tracking technology; an activity map generation unit for analyzing user eye data obtained through the user calibration to calculate the activity of each light source according to the user's gaze and generating an activity map of each light source; a first gaze tracking unit for confirming a first gaze direction, which is a gaze direction of the user, through an image of the user's eyes captured by the camera; a control information confirmation unit for confirming control information of each of the light sources corresponding to the first gaze direction in the activity map of each of the light sources; and a light control unit for controlling each of the light sources according to the control information.

[0007] FIG. 1 is a flowchart illustrating a process of controlling a light source for gaze tracking according to one embodiment.

[0008] FIG. 2 is a drawing illustrating an example of a light source that forms a glint according to the direction of a user's gaze according to one embodiment.

[0009] FIG. 3 is a diagram illustrating an example of controlling a light source for gaze tracking according to one embodiment.

[0010] FIG. 4 is a schematic diagram illustrating a configuration of a device for controlling a light source for eye tracking according to one embodiment.

[0011] FIG. 5 is a block diagram of an electronic device within a network environment according to one embodiment.

[0012] FIG. 6 is a diagram illustrating the structure of an electronic device implemented in the form of wearable augmented reality glasses according to one embodiment.

[0013] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be modified in various ways, and the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, or alternatives to the embodiments are included within the scope of the patent application.

[0014] The terms used in the examples are for illustrative purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood to not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0015] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiments pertain. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0016] In addition, when describing with reference to the attached drawings, identical components will be assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted. When describing embodiments, if a detailed description of a related known technology is judged to unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

[0017] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the embodiments. These terms are only intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by the terms. When a component is described as being "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but another component may also be "connected," "coupled," or "connected" between each component.

[0018] Components included in one embodiment and components with common functions will be described using the same names in other embodiments. Unless otherwise stated, the descriptions given in one embodiment may also apply to other embodiments, and detailed descriptions will be omitted to the extent of overlap.

[0019] Hereinafter, a device and method for controlling a light source for gaze tracking according to an embodiment of the present invention will be described in detail with reference to the attached FIGS. 1 to 6.

[0020] FIG. 1 is a flowchart illustrating a process of controlling a light source for gaze tracking according to one embodiment.

[0021] Referring to FIG. 1, a device (hereinafter, electronic device) for controlling a light source for eye tracking of the present disclosure performs user calibration using eye tracking technology (110).

[0022] At this time, user calibration is a process performed before utilizing the user gaze tracking technology of the XR device. During the user calibration process, the electronic device uses the device's display to show the user an object (e.g., a dot, a star, etc.) that the user can focus on in one area of ​​the display, and asks the user to look at the object. Specifically, during the user calibration process, the electronic device collects the user's eye data while the user looks at the object. Then, after sufficient eye data collection has occurred, the electronic device displays an object that the user can focus on in another area of ​​the display, inducing the user to look at a new location and collecting eye data. By repeating the above process, the electronic device can collect the user's eye data as the user looks at various areas of the display.

[0023] In addition, the electronic device analyzes the user's eye data acquired through user calibration to calculate the activity of each light source according to the user's gaze, thereby generating an activation map of each light source (120). At this time, the activation map may include control information corresponding to the operation status information of each light source according to the user's gaze direction.

[0024] Here, the activity map can be expressed in matrix form.

[0025] For example, the activity map is expressed as a three-dimensional matrix, the first axis of the three-dimensional matrix represents the user's X-axis gaze direction, the second axis represents the user's Y-axis gaze direction, the third axis represents identification information that identifies a light source, and the cell values ​​of the three-dimensional matrix may be control information of the light source corresponding to the identification information under the corresponding conditions.

[0026] Meanwhile, the activity map can also be expressed as a probability distribution function.

[0027] For example, the activity map is expressed as a three-dimensional probability distribution, and the probability inputs are the user gaze direction (X-axis), the user gaze direction (Y-axis), and identification information that identifies the light source, and the probability value can be control information for the light source.

[0028] Additionally, the activity map can also be expressed as a two-dimensional probability distribution function for each light source.

[0029] And, the electronic device confirms the first gaze direction, which is the user's gaze direction confirmed through an image captured of the user's pupil (130).

[0030] Then, the electronic device checks the control information of each light source corresponding to the first gaze direction in the activation map (140). In step 140, the electronic device can check the control information of each light source in the activation map by considering not only the first gaze direction but also the user's previous gaze direction. For example, the electronic device can supplement the first gaze direction by weighting the first gaze direction and the user's previous gaze direction, and check the control information of each light source corresponding to the supplemented gaze direction in the activation map.

[0031] Meanwhile, the control information for each light source may be information for turning each light source on or off. For example, the control information for each light source may be information for turning each light source on or off, each light source being represented as 0 or 1.

[0032] On the other hand, the control information for each light source may be information indicating the brightness of each light source. For example, the control information for each light source may be information indicating the brightness value of each light source, expressed as a real number between 0 and 1.

[0033] In step 140, the electronic device may set control information to turn on each of the light sources at a preset brightness if the first gaze direction is not detected. In this case, the preset brightness may be the brightness required to capture the user's pupils using a camera to detect the user's gaze direction.

[0034] And, the electronic device controls each light source according to the control information (150).

[0035] The reason for controlling the light sources according to the user's gaze through steps 130 to 150 is explained below through Figures 2 and 3.

[0036] FIG. 2 is a drawing illustrating an example of a light source that forms a glint according to the direction of a user's gaze according to one embodiment.

[0037] Referring to FIG. 2, (a) when it is confirmed that the user's eyes are facing the right through the camera (210), the first light source (220) does not form a glint because the light does not reach the pupil, but the second light source (230) can form a glint because the light reaches the pupil.

[0038] (b) When it is confirmed through the camera (210) that the user's eyes are facing the left, it can be confirmed that the light from the first light source (220) reaches the pupil and forms a glint, but the light from the second light source (230) does not reach the pupil and thus does not form a glint.

[0039] The first light source (220) in (a) and the second light source (230) in (b) do not form glints, so they are unnecessary light and can actually interfere with other light sources that form glints. Therefore, it would be desirable to turn off or minimize the first light source (220) in (a) and the second light source (230) in (b).

[0040] FIG. 3 is a diagram illustrating an example of controlling a light source for gaze tracking according to one embodiment.

[0041] Referring to FIG. 3, in step 310, the electronic device can check the user's gaze direction through the camera (210).

[0042] And, in step 320, the electronic device can check the control information of each of the first light source (220) and the second light source (230) using the activity map (322) corresponding to the first light source (220) and the activity map (324) corresponding to the second light source (230).

[0043] And, in step 330, the electronic device can adjust the first light source (220) to increase brightness and the second light source (230) to decrease brightness according to the control information confirmed in step 320.

[0044] And, the electronic device can identify the second gaze direction, which is the user's gaze direction, by using the glint generated in the user's eyes by the light sources controlled according to the control information (160).

[0045] And, the electronic device can determine the user's final gaze direction by considering the first gaze direction and the second gaze direction (170).

[0046] Then, the electronic device checks whether gaze tracking has ended (180).

[0047] If the gaze tracking has not been terminated as a result of the 180-step verification, the electronic device can return to step 130 and repeat the subsequent steps.

[0048]

[0049] FIG. 4 is a schematic diagram illustrating a configuration of a device for controlling a light source for eye tracking according to one embodiment.

[0050] Referring to FIG. 4, a device for controlling a light source for eye tracking (hereinafter, electronic device (400)) may be configured to include a processor (410), a display unit (420), a memory (430), a light control unit (440), and an eye tracking sensor unit (450).

[0051] The processor (410) may include a calibration processing unit (411), an activity map generation unit (412), a first gaze tracking unit (413), a control information verification unit (414), a second gaze tracking unit (415), and a gaze determination unit (416).

[0052] The calibration processing unit (411) can perform user calibration using eye tracking technology.

[0053] More specifically, the calibration processing unit (411) may show the user an object (e.g., a dot, a shape, etc.) on which the user can focus in one area of ​​the display using the display unit (420) of the device, and may request the user to look at the object on which the user can focus. Then, the calibration processing unit (411) may collect user eye data while the user looks at the object. Then, after sufficient eye data collection has been completed, the calibration processing unit (411) may display an object on which the user can focus in another area of ​​the display again, thereby inducing the user to look at a new location, thereby collecting eye data. The calibration processing unit (411) may repeatedly perform this process to collect user eye data when the user looks at various areas of the display.

[0054] The activity map generation unit (412) can analyze the user eye data acquired through user calibration, calculate the activity of each light source according to the user's gaze, and generate an activity map of each light source. At this time, the activity map can include control information corresponding to the operation status information of each light source according to the user's gaze direction.

[0055] Here, the activity map can be expressed in matrix form.

[0056] For example, the activity map is expressed as a three-dimensional matrix, the first axis of the three-dimensional matrix represents the user's X-axis gaze direction, the second axis represents the user's Y-axis gaze direction, the third axis represents identification information that identifies a light source, and the cell values ​​of the three-dimensional matrix may be control information of the light source corresponding to the identification information under the corresponding conditions.

[0057] Meanwhile, the activity map can also be expressed as a probability distribution function.

[0058] For example, the activity map is expressed as a three-dimensional probability distribution, and the probability inputs are the user gaze direction (X-axis), the user gaze direction (Y-axis), and identification information that identifies the light source, and the probability value can be control information for the corresponding light source.

[0059] Additionally, the activity map can also be expressed as a two-dimensional probability distribution function for each light source.

[0060] The first gaze tracking unit (413) can analyze the user's eye image captured through the camera unit (451) to confirm the first gaze direction, which is the user's gaze direction.

[0061] The control information verification unit (414) can verify the control information of each light source corresponding to the first viewing direction in the activity map of each light source.

[0062] The control information verification unit (414) can verify control information for each light source in the activity map by considering not only the first gaze direction but also the user's previous gaze direction. For example, the electronic device can supplement the first gaze direction by weighting the first gaze direction and the user's previous gaze direction, and verify control information for each light source corresponding to the supplemented gaze direction in the activation map.

[0063] At this time, the control information for each light source may be information for turning each light source on or off. For example, the control information for each light source may be information for turning each light source on or off, each light source being indicated as 0 or 1.

[0064] On the other hand, the control information for each light source may be information indicating the brightness of each light source. For example, the control information for each light source may be information indicating the brightness value of each light source, expressed as a real number between 0 and 1.

[0065] The control information verification unit (414) can set control information to turn on each light source at a preset brightness if the first gaze direction is not detected. In this case, the preset brightness may be the minimum brightness required to capture the user's pupils using the camera unit (451) to detect the user's gaze direction.

[0066] The second gaze tracking unit (415) can identify the second gaze direction, which is the direction of the user's gaze, by using glints generated in the user's eyes by light sources controlled according to control information.

[0067] The gaze determination unit (416) can determine the user's final gaze direction by considering the first gaze direction and the second gaze direction. For example, the gaze determination unit (416) can determine the final gaze direction by taking a weighted average of the first gaze direction and the second gaze direction.

[0068] The display unit (420) displays status information (or indicators), limited numbers and characters, moving pictures, and still pictures generated during the operation of the electronic device (400). In addition, according to the present disclosure, an object (e.g., a point, a transforming object, etc.) on which a user can focus can be displayed under the control of the calibration processing unit (411) for user calibration.

[0069] The memory (430) stores an operating system, application programs, and storage data for controlling the overall operation of the electronic device (400). In addition, the memory (430) can store an activity map corresponding to each of the plurality of light sources (452, 453) generated by the activity map generation unit (412) according to the present disclosure.

[0070] The light control unit (440) can control the on / off or brightness of each of the plurality of light sources (452, 453) according to the control information confirmed by the control information confirmation unit (414). In FIG. 4, the light control unit (440) is implemented as a separate device, but it can also be included in the processor (410) and operated.

[0071] The gaze tracking sensor unit (450) may include a camera unit (451) and multiple light sources (452, 453).

[0072] The camera unit (451) can capture the user's eyes to obtain and track the user's gaze direction. At this time, the camera unit (451) may be an infrared camera.

[0073] A plurality of light sources (452, 453) can turn light on / off and adjust the brightness of light under the control of a light control unit (440) to obtain information about the user's eyes. At this time, the plurality of light sources (452, 453) may be infrared light sources and may be composed of IR LEDs.

[0074]

[0075] Meanwhile, the electronic device (400) of FIG. 4 may be configured in the form of an electronic device (301) in a network environment as in FIG. 5 below, or may be configured in the form of wearable augmented reality glasses (600) as in FIG. 6.

[0076] FIG. 5 is a block diagram of an electronic device (501) within a network environment (500) according to one embodiment.

[0077] Referring to FIG. 5, in a network environment (500), an electronic device (501) may communicate with an electronic device (502) via a first network (598) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (504) or a server (508) via a second network (599) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (501) may communicate with the electronic device (504) via the server (508). According to one embodiment, the electronic device (501) may include a processor (520), a memory (530), an input module (550), an audio output module (555), a display module (560), an audio module (570), a sensor module (576), an interface (577), a connection terminal (578), a haptic module (579), a power management module (588), a battery (589), a communication module (590), a subscriber identification module (596), or an antenna module (597). In some embodiments, the electronic device (501) may omit at least one of these components (e.g., the connection terminal (578)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (576), the camera module (580), or the antenna module (597)) may be integrated into one component (e.g., the display module (560)).

[0078] The processor (520) may control at least one other component (e.g., a hardware or software component) of the electronic device (501) connected to the processor (520) by executing, for example, software (e.g., a program (540)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (520) may store a command or data received from another component (e.g., a sensor module (576) or a communication module (590)) in a volatile memory (532), process the command or data stored in the volatile memory (532), and store the resulting data in a non-volatile memory (534). According to one embodiment, the processor (520) may include a main processor (521) (e.g., a central processing unit or an application processor) or a secondary processor (523) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (501) includes a main processor (521) and a secondary processor (523), the secondary processor (523) may be configured to use less power than the main processor (521) or to be specialized for a given function. The secondary processor (523) may be implemented separately from the main processor (521) or as a part thereof.

[0079] The auxiliary processor (523) may control at least a portion of functions or states associated with at least one component (e.g., a display module (560), a sensor module (576), or a communication module (590)) of the electronic device (501), for example, on behalf of the main processor (521) while the main processor (521) is in an inactive (e.g., sleep) state, or together with the main processor (521) while the main processor (521) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (523) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (580) or a communication module (590)). In one embodiment, the auxiliary processor (523) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (501) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (508)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0080] Meanwhile, the processor (520) can perform the operation of the processor (410) of FIG. 4.

[0081] The memory (530) can store various data used by at least one component (e.g., the processor (520) or the sensor module (576)) of the electronic device (501). The data can include, for example, software (e.g., the program (540)) and input data or output data for commands related thereto. The memory (530) can include a volatile memory (532) or a non-volatile memory (534).

[0082] The program (540) may be stored as software in the memory (530) and may include, for example, an operating system (542), middleware (544), or an application (546).

[0083] The input module (550) can receive commands or data to be used in a component of the electronic device (501) (e.g., a processor (520)) from an external source (e.g., a user) of the electronic device (501). The input module (550) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0084] The audio output module (555) can output audio signals to the outside of the electronic device (501). The audio output module (555) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0085] The display module (560) can visually provide information to an external party (e.g., a user) of the electronic device (501). The display module (560) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (560) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0086] The audio module (570) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (570) can acquire sound through the input module (550), output sound through the sound output module (555), or an external electronic device (e.g., electronic device (502)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (501).

[0087] The sensor module (576) can detect the operating status (e.g., power or temperature) of the electronic device (501) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (576) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, a Hall sensor, or an illuminance sensor.

[0088] Meanwhile, the sensor module (576) may include a camera unit (451) and multiple light sources (452, 453) of FIG. 4.

[0089] Additionally, the sensor module (576) may further include a camera module capable of capturing still images and moving images. In this case, the camera module may include one or more lenses, image sensors, image signal processors, or flashes.

[0090] The interface (577) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (501) with an external electronic device (e.g., the electronic device (502)). In one embodiment, the interface (577) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0091] For example, the electronic device (501) can transmit a video signal to an external electronic device through a connection terminal (578). The electronic device (501) can transmit a video signal for the external electronic device to output a video to a display module (560) of the external electronic device.

[0092] The connection terminal (578) may be for outputting a video signal or for outputting an audio signal. For example, the connection terminal (578) may be for outputting a video signal and an audio signal simultaneously. For example, the electronic device (501) may output a video signal and an audio signal to an external electronic device through an interface such as HDMI, DP, or Thunderbolt from the connection terminal (578) that outputs a video signal and an audio signal simultaneously.

[0093] The connection terminal (578) may include a connector through which the electronic device (501) may be physically connected to an external electronic device (e.g., the electronic device (502)). According to one embodiment, the connection terminal (578) may include, for example, an HDMI connector, a DP connector, a Thunderbolt connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0094] The haptic module (579) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (579) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0095] The power management module (588) can manage power supplied to the electronic device (501). According to one embodiment, the power management module (588) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0096] A battery (589) may power at least one component of the electronic device (501). In one embodiment, the battery (589) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0097] The communication module (590) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (501) and an external electronic device (e.g., electronic device (502), electronic device (504), or server (508)), and the performance of communication through the established communication channel. The communication module (590) may operate independently from the processor (520) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (590) may include a wireless communication module (592) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (594) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (504) via a first network (598) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (599) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (592) can verify or authenticate the electronic device (501) within a communication network such as the first network (598) or the second network (599) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (596).

[0098] The wireless communication module (592) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (592) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (592) can support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (592) can support various requirements specified in the electronic device (501), an external electronic device (e.g., the electronic device (504)), or a network system (e.g., the second network (599)). According to one embodiment, the wireless communication module (592) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0099] The antenna module (597) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (597) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (597) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (598) or the second network (599), may be selected from the plurality of antennas by, for example, the communication module (590). A signal or power may be transmitted or received between the communication module (590) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (597).

[0100] According to various embodiments, the antenna module (597) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0101] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0102] According to one embodiment, commands or data may be transmitted or received between the electronic device (501) and an external electronic device (504) via a server (508) connected to a second network (599). Each of the external electronic devices (502 or 504) may be the same or a different type of device as the electronic device (501). According to one embodiment, all or part of the operations executed in the electronic device (501) may be executed in one or more of the external electronic devices (502, 504, or 508). For example, when the electronic device (501) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (501) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (501). The electronic device (501) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (501) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (504) may include an Internet of Things (IoT) device. The server (508) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (504) or the server (508) may be included in the second network (599).The electronic device (501) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0103]

[0104] FIG. 6 is a diagram illustrating the structure of an electronic device implemented in the form of wearable augmented reality glasses according to one embodiment.

[0105] Referring to FIG. 6, an electronic device (600) may be worn on a user's face and may provide the user with images related to an augmented reality service and / or a virtual reality service.

[0106] In one embodiment, the electronic device (600) includes a first display (605), a second display (610), a first screen display unit (615a), a second screen display unit (615b), an input optical member (620), a first transparent member (625a), a second transparent member (625b), a lighting member (630a, 630b), a first PCB (635a), a second PCB (635b), a first hinge (640a), a second hinge (640b), a first camera (645a, 645b, 645c, 645d), a plurality of microphones (e.g., a first microphone (650a), a second microphone (650b), a third microphone (650c)), a plurality of speakers (e.g., a first speaker (655a), a second speaker (655b)), a battery (660), a second camera (675a, 675b), a third camera (665), a visor (670a, 670b), a right eye light source (641, 642, 643, 661, 662, 663), and a left eye light source (651, 652, 653, 671, 672, 673).

[0107] At this time, the second camera (675a, 675b) corresponds to the camera unit (451) of FIG. 4, and the right eye light sources (641, 642, 643, 661, 662, 663) and the left eye light sources (651, 652, 653, 671, 672, 673) can correspond to the multiple light sources (452, 453) of FIG. 4.

[0108] In one embodiment, the displays (e.g., the first display (605) and the second display (610)) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not shown, when the display is comprised of one of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the electronic device (600) may include a light source that irradiates light onto a screen output area of ​​the display. In another embodiment, when the display is capable of generating light on its own, for example, comprised of one of an organic light emitting diode or a micro LED, the electronic device (600) may provide a good quality virtual image to the user even without including a separate light source. In one embodiment, if the display is implemented with an organic light-emitting diode or micro LED, a light source is unnecessary, so the electronic device (600) can be made lighter. In the following, a display that can generate light on its own is referred to as a self-luminous display, and the description is based on the assumption of a self-luminous display.

[0109] The displays according to various embodiments of the present document (e.g., the first display (605) and the second display (610)) may be composed of at least one micro LED (micro light emitting diode). For example, the micro LED can express red (R, red), green (G, green), and blue (B, blue) by self-luminescence, and is small in size (e.g., 100 μm or less), so that one chip can implement one pixel (e.g., one of R, G, and B). Accordingly, when the display is composed of the micro LED, it can provide a high resolution without a backlight unit (BLU).

[0110] Not limited thereto, one pixel may include R, G, and B, and one chip may be implemented with multiple pixels including R, G, and B.

[0111] In one embodiment, the display (e.g., the first display (605) and the second display (610)) may be configured with a display area composed of pixels for displaying a virtual image and light-receiving pixels (e.g., photo sensor pixels) arranged between the pixels for receiving light reflected from the eye, converting it into electrical energy, and outputting it.

[0112] In one embodiment, the electronic device (600) can detect the direction of the user's gaze (e.g., eye movement) through light-receiving pixels. For example, the electronic device (600) can detect and track the direction of the user's gaze with respect to the right eye and the direction of the user's gaze with respect to the left eye through one or more light-receiving pixels constituting the first display (605) and one or more light-receiving pixels constituting the second display (610). The electronic device (600) can determine the position of the center of the virtual image based on the direction of the user's gaze with respect to the right and left eyes (e.g., the direction in which the pupils of the user's right and left eyes are gazing) detected through one or more light-receiving pixels.

[0113] In one embodiment, the odd-numbered light sources (641, 642, 643) for the right eye, the right eye light sources (641, 642, 643, 661, 662, 663) and the left eye light sources (651, 652, 653, 671, 672, 673) attached around the frame of the electronic device (600) can be used as auxiliary means to facilitate detection of eye gaze when photographing the pupil with the second camera (675a, 675b). When the right eye light source (641, 642, 643, 661, 662, 663) and the left eye light source (651, 652, 653, 671, 672, 673) are used as auxiliary means for detecting the direction of gaze, they may be composed of light emitting diodes (LEDs) or infrared ray light emitting diodes (IR LEDs) that generate infrared wavelengths.

[0114] In one embodiment, light emitted from displays (e.g., the first display (605) and the second display (610)) may pass through a lens (not shown) and a waveguide to reach a first screen display unit (615a) formed on a first transparent member (625a) positioned to face the user's right eye and a second screen display unit (615b) formed on a second transparent member (625b) positioned to face the user's left eye. For example, light emitted from displays (e.g., the first display (605) and the second display (610)) may pass through a waveguide to be reflected by a grating area formed on an input optical member (620) and the screen display units (615a, 615b) and transmitted to the user's eyes. The first transparent member (625a) and / or the second transparent member (625b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent.

[0115] In one embodiment, a lens (not shown) may be positioned in front of a display (e.g., a first display (605) and a second display (610)). The lens (not shown) may include a concave lens and / or a convex lens. For example, the lens (not shown) may include a projection lens or a collimation lens.

[0116] In one embodiment, the screen display (615a, 615b) or the transparent member (e.g., the first transparent member (625a), the second transparent member (625b)) may include a lens including a waveguide, a reflective lens.

[0117] In one embodiment, the waveguide may be made of glass, plastic, or polymer, and may include nano-patterns formed on one surface of the inner or outer surface, for example, a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one end of the waveguide may be propagated within the display waveguide by the nano-patterns and provided to the user. In one embodiment, a waveguide composed of a free-form prism may provide the incident light to the user through a reflective mirror. The waveguide may include at least one diffractive element, for example, a diffractive optical element (DOE), a holographic optical element (HOE), or a reflective element, for example, a reflective mirror. In one embodiment, the waveguide may guide light emitted from the display (605, 610) to the user's eyes by using at least one diffractive element or reflective element included in the waveguide.

[0118] According to various embodiments, the diffractive element may include an input optical member (620) / output optical member (not shown). For example, the input optical member (620) may mean an input grating area, and the output optical member (not shown) may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from a display (e.g., a first display (605) and a second display (610)) (e.g., a micro LED) to transmit the light to a transparent member (e.g., a first transparent member (650a), a second transparent member (650b)) of the screen display unit (615a, 615b). The output grating area may serve as an outlet that diffracts (or reflects) light transmitted to a transparent member (e.g., a first transparent member (650a), a second transparent member (650b)) of the waveguide to a user's eye.

[0119] According to various embodiments, the reflective element may include a total internal reflection (TIR) ​​optical element or a total internal reflection waveguide. For example, total internal reflection may refer to a method of guiding light such that light (e.g., a virtual image) entering through an input grating region is 100% reflected from one surface (e.g., a specific surface) of the waveguide, thereby transmitting 100% of the light to the output grating region.

[0120] In one embodiment, light emitted from a display (605, 610) may be guided along an optical path through an input optical member (620) into a waveguide. Light traveling within the waveguide may be guided toward a user's eyes through an output optical member. The screen display (615a, 615b) may be determined based on the light emitted toward the user's eyes.

[0121] In one embodiment, the first camera (645a, 645b, 645c, 645d) may include a camera used for 6 degrees of freedom (6DoF), 6DoF head tracking, hand detection and tracking, gesture, and / or spatial recognition. For example, the first camera (645a, 645b, 645c, 645d) may include a global shutter (GS) camera to detect and track head and hand movements.

[0122] For example, the first camera (645a, 645b, 645c, 645d) may be a stereo camera for head tracking and spatial recognition, and cameras of the same specifications and performance may be used. The first camera (645a, 645b, 645c, 645d) may be a GS camera with excellent performance (e.g., image drag) for detecting and tracking fine movements such as rapid hand movements and fingers.

[0123] According to various embodiments, the first camera (645a, 645b, 645c, 645d) may be a rolling shutter (RS) camera. The first camera (645a, 645b, 645c, 645d) may perform SLAM functions through spatial recognition and depth shooting for 6 DoF. The first camera (645a, 645b, 645c, 645d) may perform a user gesture recognition function.

[0124] In one embodiment, the second cameras (675a, 675b) may be used to detect and track the pupils. The second cameras (675a, 675b) may be referred to as ET (eye tracking) cameras. The second cameras (675a, 675b) may track the user's gaze direction. The electronic device (600) may consider the user's gaze direction and position the center of the virtual image projected on the screen display unit (615a, 615b) according to the direction in which the user's pupils are looking.

[0125] The second camera (675a, 675b) for tracking the gaze direction may be a GS camera that can detect the pupil and track rapid eye movements. The second cameras (675a, 675b) may be installed for the left and right eyes, respectively, and the second cameras (675a, 675b) for the left and right eyes may be cameras with identical performance and specifications.

[0126] In one embodiment, the third camera (665) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. The third camera (665) may include a color camera equipped with functions for obtaining high-quality images, such as an auto focus (AF) function and an optical image stabilizer (OIS). However, the third camera (665) is not limited thereto, and may include a GS (global shutter) camera or an RS (rolling shutter) camera.

[0127] In one embodiment, at least one sensor (e.g., a gyroscope sensor, an acceleration sensor, a magnetometer sensor, a touch sensor, an ambient light sensor, and / or a gesture sensor), and a first camera (645a, 645b, 645c, 645d) can perform at least one of head tracking for 6DoF, pose estimation & prediction, gesture and / or spatial recognition, and slam function with depth shooting.

[0128] In another embodiment, the first camera (645a, 645b, 645c, 645d) may be used separately as a camera for head tracking and a camera for hand tracking.

[0129] In one embodiment, the lighting units (630a, 630b) may have different uses depending on the attachment location. For example, the lighting units (630a, 630b) may be attached together with the first camera (645a, 645b, 645c, 645d) mounted around a hinge (e.g., a first hinge (640a), a second hinge (640b)) connecting a frame and a temple or around a bridge connecting the frames. When shooting with a GS camera, the lighting units (630a, 630b) may be used as a means of supplementing the ambient brightness. For example, the lighting units (630a, 630b) may be used when it is difficult to detect a subject to be shot due to a dark environment or mixing or reflecting light from multiple light sources.

[0130] In one embodiment, a PCB (e.g., a first PCB (635a), a second PCB (635b)) may include a processor (not shown), a memory (not shown), and a communication module (not shown) that control components of the electronic device (600).

[0131] A communication module (not shown) may support the establishment of a direct (e.g., wired) or wireless communication channel between the electronic device (600) and an external electronic device, and the performance of communication through the established communication channel. The PCB may transmit electrical signals to components constituting the electronic device (600).

[0132] A communication module (not shown) may include one or more communication processors that operate independently from the processor and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (not shown) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules (not shown) may communicate with an external electronic device via a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0133] The wireless communication module can support 5G networks and next-generation communication technologies beyond 4G networks, such as new radio access technology (NR). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module can support high-frequency bands (e.g., mmWave bands) to achieve high data rates, for example. The wireless communication module may support various technologies to secure performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna.

[0134] The electronic device (600) may further include an antenna module (not shown). The antenna module may transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a first PCB (635a), a second PCB (635b)). In one embodiment, the antenna module may include a plurality of antennas (e.g., an array antenna).

[0135] In one embodiment, multiple microphones (e.g., a first microphone (650a), a second microphone (650b), and a third microphone (650c)) can process external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function being performed (or the application being executed) by the electronic device (600).

[0136] In one embodiment, multiple speakers (e.g., first speaker (655a), second speaker (655b)) can output audio data received from a communication module or stored in a memory.

[0137] In one embodiment, one or more batteries (660) may be included and may supply power to components that make up the electronic device (600).

[0138] In one embodiment, the visor (670a, 670b) can adjust the amount of external light transmitted into the user's eyes according to the transmittance. The visor (670a, 670b) can be positioned in front or behind the screen display unit (615a, 615b). The front of the screen display unit (615a, 615b) can mean the direction opposite to the user wearing the electronic device (600), and the back can mean the direction toward the user wearing the electronic device (600). The visor (670a, 670b) can protect the screen display unit (615a, 615b) and adjust the amount of external light transmitted.

[0139] For example, the visor (670a, 670b) may include an electrochromic element that changes color according to the applied power to control the transmittance. Electrochromism is a phenomenon in which the color changes due to an oxidation-reduction reaction caused by the applied power. The visor (670a, 670b) can control the transmittance of external light by utilizing the color change of the electrochromic element.

[0140] For example, the visor (670a, 670b) may include a control module and an electrochromic element. The control module may control the electrochromic element to adjust the transmittance of the electrochromic element.

[0141]

[0142] According to one embodiment, a method for controlling a light source for gaze tracking may include: performing user calibration using gaze tracking technology; analyzing user eye data acquired through the user calibration to calculate the activity of each light source according to the gaze of the user, thereby generating an activity map of each light source; confirming a first gaze direction, which is a gaze direction of the user confirmed through an image capturing the pupil of the user; confirming control information of each light source corresponding to the first gaze direction in the activity map of each light source; and controlling each light source according to the control information.

[0143] According to one embodiment, the activity map may include control information for each of the light sources according to the user's gaze direction.

[0144] According to one embodiment, the activity map is expressed as a three-dimensional matrix, a first axis of the three-dimensional matrix represents the user's X-axis gaze direction, a second axis represents the user's Y-axis gaze direction, a third axis represents identification information for identifying a light source, and a cell value of the three-dimensional matrix may be control information for a light source corresponding to the identification information under a given condition.

[0145] According to one embodiment, the operation of checking the control information of each of the light sources may check the control information of each of the light sources in the activity map by taking into consideration the user's previous gaze direction and the first gaze direction.

[0146] According to one embodiment, the control information for each of the light sources may be information for turning each of the light sources on or off.

[0147] According to one embodiment, the control information for each of the light sources may be information for turning on or off each of the light sources, each of which is indicated by 0 or 1.

[0148] According to one embodiment, the control information for each of the light sources may be information indicating the brightness of each of the light sources.

[0149] According to one embodiment, the control information for each of the light sources may be information indicating the brightness value of each of the light sources, which is expressed as a real number between 0 and 1.

[0150] According to one embodiment, the operation of checking the control information of each of the light sources may be set to control information that turns on each of the light sources at a preset brightness if the first gaze direction is not detected.

[0151] In one embodiment, the preset brightness may be a brightness required to capture the user's pupils using a camera to detect the user's gaze direction.

[0152] According to one embodiment, a method for controlling a light source for eye tracking may further include an operation of identifying a second gaze direction, which is a gaze direction of the user, by using a glint generated in the user's eyes by the controlled light sources according to the control information; and an operation of determining a final gaze direction of the user by considering the first gaze direction and the second gaze direction.

[0153] According to one embodiment, a device for controlling a light source for eye tracking may include: a camera for photographing a user's eyes; light sources for obtaining information about the user's eyes; a calibration processing unit for performing user calibration using eye tracking technology; an activity map generation unit for analyzing user eye data obtained through the user calibration to calculate the activity of each light source according to the user's gaze and generating an activity map of each light source; a first gaze tracking unit for confirming a first gaze direction, which is a gaze direction of the user, through an image of the user's eyes captured by the camera; a control information confirmation unit for confirming control information of each of the light sources corresponding to the first gaze direction in the activity map of each of the light sources; and a light control unit for controlling each of the light sources according to the control information.

[0154] According to one embodiment, the activity map may include control information for each of the light sources according to the user's gaze direction.

[0155] According to one embodiment, the activity map is expressed as a three-dimensional matrix, a first axis of the three-dimensional matrix represents the user's X-axis gaze direction, a second axis represents the user's Y-axis gaze direction, a third axis represents identification information for identifying a light source, and a cell value of the three-dimensional matrix may be control information for a light source corresponding to the identification information under a given condition.

[0156] According to one embodiment, the control information verification unit can verify control information of each of the light sources in the activity map by considering the user's current gaze direction and the user's previous gaze direction.

[0157] According to one embodiment, the control information for each of the light sources may be information for turning each of the light sources on or off, or information indicating the brightness of each of the light sources.

[0158] According to one embodiment, the control information verification unit may set control information to turn on each of the light sources at a preset brightness if the user's current gaze direction is not detected.

[0159] In one embodiment, the preset brightness may be a brightness required to capture the user's pupils using a camera to detect the user's gaze direction.

[0160] According to one embodiment, a device for controlling a light source for eye tracking may further include a second eye tracking unit that determines a second eye direction, which is a direction of the user's eye gaze, by using a glint generated in the user's eye by the controlled light sources according to the control information; and a gaze determining unit that determines a final eye direction of the user by considering the first eye direction and the second eye direction.

[0161]

[0162] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., singly or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0163] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems, and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0164] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the above. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0165] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A method for controlling a light source for gaze tracking, Actions that perform user calibration using eye tracking technology; An operation of analyzing the user eye data acquired through the above user calibration, calculating the activity of each light source according to the user's gaze, and generating an activity map of each light source; An action of confirming the first gaze direction, which is the gaze direction of the user confirmed through an image captured of the user's pupil; An operation of checking control information of each of the light sources corresponding to the first gaze direction in the activity map of each of the light sources; and An operation for controlling each of the light sources according to the above control information. A method including:

2. In paragraph 1, The above activity map is, Contains control information for each of the light sources according to the user's gaze direction. method.

3. In any one of paragraphs 1 and 2, The above activity map is, It is expressed as a three-dimensional matrix, the first axis of the three-dimensional matrix represents the X-axis gaze direction of the user, the second axis represents the Y-axis gaze direction of the user, the third axis represents identification information that identifies the light source, and the cell value of the three-dimensional matrix is ​​control information for the light source corresponding to the identification information under the corresponding condition. method.

4. In any one of paragraphs 1 to 3, The operation of checking the control information of each of the above light sources is as follows: Checking the control information of each of the light sources in the activity map by considering the previous gaze direction of the user and the first gaze direction. method.

5. In any one of paragraphs 1 to 4, The control information for each of the above light sources is: Information to turn each of the above light sources on or off method.

6. In any one of paragraphs 1 to 5, The control information for each of the above light sources is: Information for turning on or off each of the above light sources, each of which is indicated as 0 or 1. method.

7. In any one of paragraphs 1 to 6, The control information for each of the above light sources is: Information indicating the brightness of each of the above light sources method.

8. In any one of paragraphs 1 to 7, The control information for each of the above light sources is: Information indicating the brightness value of each of the above light sources, which is expressed as a real number between 0 and 1. method 9. In any one of paragraphs 1 to 8, The operation of checking the control information of each of the above light sources is as follows: If the above first gaze direction is not detected, control information is set to turn on each of the light sources at a preset brightness. method.

10. In any one of paragraphs 1 to 9, The above preset brightness is, The brightness required to capture the user's pupils using a camera to detect the user's gaze direction method.

11. In any one of paragraphs 1 to 10, An operation of confirming a second gaze direction, which is the gaze direction of the user, by using a glint generated in the user's eyes by the controlled light sources according to the above control information; and An operation for determining the final gaze direction of the user by considering the first gaze direction and the second gaze direction. How to include more.

12. In a device for controlling a light source for gaze tracking, A camera that photographs the user's eyes; Light sources for obtaining eye information of the user; A calibration processing unit that performs user calibration using eye tracking technology; An activity map generation unit that analyzes user eye data acquired through the above user calibration, calculates the activity of each light source according to the user's gaze, and generates an activity map of each light source; A first gaze tracking unit that confirms the first gaze direction, which is the gaze direction of the user, through an image of the user's eye captured by the camera; A control information verification unit for verifying control information of each of the light sources corresponding to the first gaze direction in the activity map of each of the light sources; and A light control unit that controls each of the light sources according to the above control information. A device comprising:

13. In paragraph 12, The above activity map is, Contains control information for each of the light sources according to the user's gaze direction. device.

14. In any one of paragraphs 12 to 13, The above activity map is, It is expressed as a three-dimensional matrix, the first axis of the three-dimensional matrix represents the X-axis gaze direction of the user, the second axis represents the Y-axis gaze direction of the user, the third axis represents identification information that identifies the light source, and the cell value of the three-dimensional matrix is ​​control information for the light source corresponding to the identification information under the corresponding condition. device.

15. In any one of paragraphs 12 to 14, The above control information verification unit, Check the control information of each light source in the activity map by considering the current gaze direction of the user and the previous gaze direction of the user. device.

Images