Electronic device and method for acquiring user's gaze information
Patent Information
- Application Number
- IN202337010153
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-09-03
- Filing Date
- 2023-02-15
- Publication Date
- 2026-10-27
- Estimated Expiration
- 2041-08-19
AI Technical Summary
Existing electronic devices that detect user gaze information require separate infrared cameras and optical instruments, making them difficult to miniaturize and increasing power consumption.
Incorporating light-receiving pixels into the display of electronic devices, such as those composed of micro LEDs, to detect gaze information without the need for separate optical instruments, allowing the device to miniaturize and reduce power consumption.
Enables the detection of user gaze information using light-receiving pixels in the display, facilitating miniaturization and reducing power consumption by eliminating the need for separate optical instruments.
Abstract
Description
[Technical Field]
[0001] The disclosure relates to an electronic device and a method of acquiringgaze information of a user.[Background Art]
[0002] An electronic device such as an augmented reality (AR) device mayprovide a user with an image, for example, a virtual image based on the real world.The electronic device may output an image based on the gaze information of the user,and may detect and track the movement of eyes, for example, pupils of the userwearing the electronic device by using, for example, an infrared camera. For instance,the electronic device may output an optical signal through an optical instrument suchas an infrared light emitting diodes (LED). The output light may be reflected from theuser's eyes. The electronic device may receive reflected light through an image sensorof the infrared camera, and detect and track eye movements based on the receivedlight. The electronic device may output, based on the position and movement directionof the user's eyes, a virtual image corresponding to the direction in which the user'seyes move.[Disclosure of Invention][Technical Problem]
[0003] However, since such an electronic device has to separately equippedwith an infrared camera as well as an optical instrument to obtain the user's gazeinformation, it is not only difficult to miniaturize the electronic device, but also powerconsumption may be high because the optical instrument and the infrared cameramust be simultaneously driven.
[0004] The electronic device includes a display composed of micro LEDs. Thedisplay composed of micro LEDs may include a plurality of pixels for displaying animage and light-receiving pixels arranged between the plurality of pixels.
[0005] According to various embodiments of the disclosure, light output fromthe display composed of micro LEDs may be reflected from the user's eyes andreceived by one or more light-receiving pixels constituting the display. The electronicdevice may detect and track the gaze of the user based on light received through oneor more light-receiving pixels.[Solution to Problem]
[0006] In accordance with another aspect of the disclosure, an electronic deviceis provided. The electronic device includes a plurality of pixels, light-receiving pixelsarranged between the plurality of pixels, a display composed of the plurality of pixelsand the light-receiving pixels, and a processor operatively connected to the display,wherein the processor may be configured to receive light reflected from each of user'sleft and right eyes through one or more light-receiving pixels constituting the display,identify a gaze direction of each of the left and right eyes based on the light receivedthrough the one or more light-receiving pixels, and output an image based on theidentified gaze direction.
[0007] In accordance with another aspect of the disclosure, a method forobtaining user's gaze information by using light-receiving pixels of a display of anelectronic device is provided. The method includes receiving light reflected from eachof user's left and right eyes through one or more light-receiving pixels constituting thedisplay, identifying a gaze direction of each of the left and right eyes based on the lightreceived through the one or more light-receiving pixels, and outputting an imagebased on the identified gaze direction.[Advantageous Effects of Invention]
[0008] According to various embodiments of the disclosure, the electronicdevice may obtain information on the gaze of the user by using one or more light-receiving pixels of the display composed of micro LEDs without being separatelyequipped with an optical instrument or an infrared camera for detecting the gaze ofthe user. As an optical instrument and an infrared camera for detecting a user's gazeare not separately provided, the electronic device can be miniaturized and lightweight.[Brief Description of Drawings]
[0009] FIG. 1 is a block diagram of an electronic device in a networkenvironment according to an embodiment of the disclosure;
[0010] FIG. 2 is a diagram illustrating an electronic device according to anembodiment of the disclosure;
[0011] FIG. 3 is a diagram illustrating an electronic device, for example, ARglasses according to an embodiment of the disclosure;
[0012] FIG. 4A is a diagram illustrating a plan view of pixels constituting thedisplay according to an embodiment of the disclosure;
[0013] FIG. 4B is a diagram illustrating an arrangement structure of pixelsconstituting the display according to an embodiment of the disclosure;
[0014] FIG. 4C is a diagram illustrating a cross-sectional view of a plurality ofpixels constituting the display according to an embodiment of the disclosure;
[0015] FIGS. 5A and 5B are diagrams for depicting a path of visible light outputfrom the display and a path of visible light reflected from a user's eye according tovarious embodiments of the disclosure;
[0016] FIG. 6 is a diagram for depicting paths of visible light and infrared lightoutput from the display according to an embodiment of the disclosure;
[0017] FIG. 7 is a block diagram illustrating a display according to anembodiment of the disclosure;
[0018] FIG. 8 is a flowchart illustrating a method of detecting a user's gazedirection and outputting an image based on the detected gaze direction according toan embodiment of the disclosure;
[0019] FIG. 9 is a flowchart illustrating a method for the electronic device tooutput an image by adjusting the periodicity of outputting a video signal and theperiodicity of receiving reflected light according to an embodiment of the disclosure;and
[0020] FIG. 10 is a diagram for depicting the periodicity of outputting a videosignal and the periodicity of receiving reflected light in the electronic device accordingto embodiment of the disclosure.[Mode for the Invention]
[0021] FIG. 1 is a block diagram illustrating an electronic device in a networkenvironment according to an embodiment of the disclosure.
[0022] Referring to FIG. 1, an electronic device 101 in a network environment100 may communicate with an electronic device 102 via a first network 198 (e.g., ashort-range wireless communication network), or at least one of an electronic device104 or a server 108 via a second network 199 (e.g., a long-range wirelesscommunication network). According to an embodiment, the electronic device 101 maycommunicate with the electronic device 104 via the server 108. According to anembodiment, the electronic device 101 may include a processor 120, memory 130, aninput module 150, a sound output module 155, a display module 160, an audio module170, a sensor module 176, an interface 177, a connection terminal 178, a hapticmodule 179, a camera module 180, a power management module 188, a battery 189,a communication module 190, a subscriber identification module (SIM) 196, or anantenna module 197. In some embodiments, at least one of the components (e.g., theconnection terminal 178) may be omitted from the electronic device 101, or one ormore other components may be added in the electronic device 101. In someembodiments, some of the components (e.g., the sensor module 176, the cameramodule 180, or the antenna module 197) may be implemented as a single component(e.g., the display module 160).
[0023] The processor 120 may execute, for example, software (e.g., a program140) to control at least one other component (e.g., a hardware or softwarecomponent) of the electronic device 101 coupled with the processor 120, and mayperform various data processing or computation. According to one embodiment, as atleast part of the data processing or computation, the processor 120 may store acommand or data received from another component (e.g., the sensor module 176 orthe communication module 190) in volatile memory 132, process the command or thedata stored in the volatile memory 132, and store resulting data in non-volatilememory 134. According to an embodiment, the processor 120 may include a mainprocessor 121 (e.g., a central processing unit (CPU) or an application processor (AP)),or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neuralprocessing unit (NPU), an image signal processor (ISP), a sensor hub processor, or acommunication processor (CP)) that is operable independently from, or in conjunctionwith, the main processor 121. For example, when the electronic device 101 includesthe main processor 121 and the auxiliary processor 123, the auxiliary processor 123may be adapted to consume less power than the main processor 121, or to be specificto a specified function. The auxiliary processor 123 may be implemented as separatefrom, or as part of the main processor 121.
[0024] The auxiliary processor 123 may control at least some of functions orstates related to at least one component (e.g., the display module 160, the sensormodule 176, or the communication module 190) among the components of theelectronic device 101, instead of the main processor 121 while the main processor 121is in an inactive (e.g., sleep) state, or together with the main processor 121 while themain processor 121 is in an active state (e.g., executing an application). According toan embodiment, the auxiliary processor 123 (e.g., an image signal processor or acommunication processor) may be implemented as part of another component (e.g.,the camera module 180 or the communication module 190) functionally related to theauxiliary processor 123. According to an embodiment, the auxiliary processor 123(e.g., the neural processing unit) may include a hardware structure specified forartificial intelligence model processing. An artificial intelligence model may begenerated by machine learning. Such learning may be performed, e.g., by theelectronic device 101 where the artificial intelligence is performed or via a separateserver (e.g., the server 108). Learning algorithms may include, but are not limited to,e.g., supervised learning, unsupervised learning, semi-supervised learning, orreinforcement learning. The artificial intelligence model may include a plurality ofartificial neural network layers. The artificial neural network may be a deep neuralnetwork (DNN), a convolutional neural network (CNN), a recurrent neural network(RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), abidirectional recurrent deep neural network (BRDNN), deep Q-network or acombination of two or more thereof but is not limited thereto. The artificialintelligence model may, additionally or alternatively, include a software structureother than the hardware structure.
[0025] The memory 130 may store various data used by at least one component(e.g., the processor 120 or the sensor module 176) of the electronic device 101. Thevarious data may include, for example, software (e.g., the program 140) and input dataor output data for a command related thereto. The memory 130 may include thevolatile memory 132 or the non-volatile memory 134. The non-volatile memory 134may include an internal memory 136 and / or an external memory 138.
[0026] The program 140 may be stored in the memory 130 as software, andmay include, for example, an operating system (OS) 142, middleware 144, or anapplication 146.
[0027] The input module 150 may receive a command or data to be used byanother component (e.g., the processor 120) of the electronic device 101, from theoutside (e.g., a user) of the electronic device 101. The input module 150 may include,for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen(e.g., a stylus pen).
[0028] The sound output module 155 may output sound signals to the outsideof the electronic device 101. The sound output module 155 may include, for example,a speaker or a receiver. The speaker may be used for general purposes, such as playingmultimedia or playing record. The receiver may be used for receiving incoming calls.According to an embodiment, the receiver may be implemented as separate from, oras part of the speaker.
[0029] The display module 160 may visually provide information to the outside(e.g., a user) of the electronic device 101. The display module 160 may include, forexample, a display, a hologram device, or a projector and control circuitry to control acorresponding one of the display, hologram device, and projector. According to anembodiment, the display module 160 may include a touch sensor adapted to detect atouch, or a pressure sensor adapted to measure the intensity of force incurred by thetouch.
[0030] The audio module 170 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 170 may obtain the soundvia the input module 150, or output the sound via the sound output module 155 or aheadphone of an external electronic device (e.g., electronic device 102) (e.g., speakeror headphone) directly (e.g., wiredly) or wirelessly coupled with the electronic device101.
[0031] The sensor module 176 may detect an operational state (e.g., power ortemperature) of the electronic device 101 or an environmental state (e.g., a state of auser) external to the electronic device 101, and then generate an electrical signal ordata value corresponding to the detected state. According to an embodiment, thesensor module 176 may include, for example, a gesture sensor, a gyro sensor, anatmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor,a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, atemperature sensor, a humidity sensor, or an illuminance sensor.
[0032] The interface 177 may support one or more specified protocols to beused for the electronic device 101 to be coupled with the external electronic device(e.g., the electronic device 102) directly (e.g., through wires) or wirelessly. Accordingto an embodiment, the interface 177 may include, for example, a high-definitionmultimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital(SD) card interface, or an audio interface.
[0033] The connection terminal 178 may include a connector via which theelectronic device 101 may be physically connected with the external electronic device(e.g., the electronic device 102). According to an embodiment, the connectionterminal 178 may include, for example, an HDMI connector, a USB connector, an SDcard connector, or an audio connector (e.g., a headphone connector).
[0034] The haptic module 179 may convert an electrical signal into a mechanicalstimulus (e.g., a vibration or a movement) or electrical stimulus which may berecognized by a user via his tactile sensation or kinesthetic sensation. According to anembodiment, the haptic module 179 may include, for example, a motor, apiezoelectric element, or an electric stimulator.
[0035] The camera module 180 may capture a still image or moving images.According to an embodiment, the camera module 180 may include one or more lenses,image sensors, image signal processors, or flashes.
[0036] The power management module 188 may manage power supplied to theelectronic device 101. According to one embodiment, the power management module188 may be implemented as at least part of, for example, a power managementintegrated circuit (PMIC).
[0037] The battery 189 may supply power to at least one component of theelectronic device 101. According to an embodiment, the battery 189 may include, forexample, a primary cell which is not rechargeable, a secondary cell which isrechargeable, or a fuel cell.
[0038] The communication module 190 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel between theelectronic device 101 and the external electronic device (e.g., the electronic device102, the electronic device 104, or the server 108) and performing communication viathe established communication channel. The communication module 190 may includeone or more communication processors that are operable independently from theprocessor 120 (e.g., an application processor (AP)) and supports a direct (e.g., wired)communication or a wireless communication. According to an embodiment, thecommunication module 190 may include a wireless communication module 192 (e.g.,a cellular communication module, a short-range wireless communication module, ora global navigation satellite system (GNSS) communication module) or a wiredcommunication module 194 (e.g., a local area network (LAN) communication moduleor a power line communication (PLC) module). A corresponding one of thesecommunication modules may communicate with the external electronic device via thefirst network 198 (e.g., a short-range communication network, such as BluetoothTM,Wi-Fi direct, or infrared data association (IrDA)) or the second network 199 (e.g., along-range communication network, such as a legacy cellular network, a fifthgeneration (5G) network, a next-generation communication network, the Internet, ora computer network (e.g., LAN or wide area network (WAN))). These various types ofcommunication modules may be implemented as a single component (e.g., a singlechip), or may be implemented as multi components (e.g., multi chips) separate fromeach other. The wireless communication module 192 may identify and authenticatethe electronic device 101 in a communication network, such as the first network 198or the second network 199, using subscriber information (e.g., international mobilesubscriber identity (IMSI)) stored in the subscriber identification module 196.
[0039] The wireless communication module 192 may support a 5th generation(5G) network, after a 4th generation (4G) network, and next-generationcommunication technology, e.g., new radio (NR) access technology. The NR accesstechnology may support enhanced mobile broadband (eMBB), massive machine typecommunications (mMTC), or ultra-reliable and low-latency communications (URLLC).The wireless communication module 192 may support a high-frequency band (e.g., themmWave band) to achieve, e.g., a high data transmission rate. The wirelesscommunication module 192 may support various technologies for securingperformance on a high-frequency band, such as, e.g., beamforming, massive multipleinput and multiple-output (massive MIMO), full dimensional MIMO (FD-MIMO), arrayantenna, analog beam-forming, or large-scale antenna. The wireless communicationmodule 192 may support various requirements specified in the electronic device 101,an external electronic device (e.g., the electronic device 104), or a network system(e.g., the second network 199). According to an embodiment, the wirelesscommunication module 192 may support a peak data rate (e.g., 20Gbps or more) forimplementing eMBB, loss coverage (e.g., 164dB or less) for implementing mMTC, orU-plane latency (e.g., 0.5ms or less for each of downlink (DL) and uplink (UL), or around trip of 1ms or less) for implementing URLLC.
[0040] The antenna module 197 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of the electronic device 101.According to an embodiment, the antenna module 197 may include an antennaincluding a radiating element composed of a conductive material or a conductivepattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According toan embodiment, the antenna module 197 may include a plurality of antennas (e.g.,array antennas). In such a case, at least one antenna appropriate for a communicationscheme used in the communication network, such as the first network 198 or thesecond network 199, may be selected, for example, by the communication module190 (e.g., the wireless communication module 192) from the plurality of antennas.The signal or the power may then be transmitted or received between thecommunication module 190 and the external electronic device via the selected at leastone antenna. According to an embodiment, another component (e.g., a radiofrequency integrated circuit (RFIC)) other than the radiating element may beadditionally formed as part of the antenna module 197.
[0041] According to various embodiments, the antenna module 197 may formmmWave antenna module. According to an embodiment, the mmWave antennamodule may include a printed circuit board, an RFIC disposed on a first surface (e.g.,the bottom surface) of the printed circuit board, or adjacent to the first surface andcapable of supporting a designated high-frequency band (e.g., an mmwave band), anda plurality of antennas (e.g., array antennas) disposed on a second surface (e.g., thetop or a side surface) of the printed circuit board, or adjacent to the second surfaceand capable of transmitting or receiving signals of the designated high-frequencyband.
[0042] At least some of the above-described components may be coupledmutually and communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purpose input and output(GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).
[0043] According to an embodiment, commands or data may be transmitted orreceived between the electronic device 101 and the external electronic device 104 viathe server 108 coupled with the second network 199. Each of the electronic devices102 or 104 may be a device of a same type as, or a different type, from the electronicdevice 101. According to an embodiment, all or some of operations to be executed atthe electronic device 101 may be executed at one or more of the external electronicdevices (e.g., electronic devices 102 and 104 or server 108). For example, if theelectronic device 101 should perform a function or a service automatically, or inresponse to a request from a user or another device, the electronic device 101, insteadof, or in addition to, executing the function or the service, may request the one ormore external electronic devices to perform at least part of the function or the service.The one or more external electronic devices receiving the request may perform the atleast part of the function or the service requested, or an additional function or anadditional service related to the request, and transfer an outcome of the performingto the electronic device 101. The electronic device 101 may provide the outcome, withor without further processing of the outcome, as at least part of a reply to the request.To that end, a cloud computing, distributed computing, mobile edge computing (MEC),or client-server computing technology may be used, for example. The electronic device101 may provide ultra-low latency services using, e.g., distributed computing or mobileedge computing. In another embodiment, the external electronic device 104 mayinclude an internet-of-things (IoT) device. The server 108 may be an intelligent serverusing machine learning and / or a neural network. According to an embodiment, theexternal electronic device 104 or the server 108 may be included in the secondnetwork 199. The electronic device 101 may be applied to intelligent services (e.g.,smart home, smart city, smart car, or healthcare) based on 5G communicationtechnology or IoT-related technology.
[0044] FIG. 2 is a diagram illustrating an electronic device according toembodiment of the disclosure.
[0045] Referring to FIG. 2, diagram 200 illustrates an electronic device 201 (e.g.,electronic device 101 in FIG. 1) including a communication circuit 210 (e.g.,communication module 190 in FIG. 1), a memory 220 (e.g., memory 130 in FIG. 1), asensor circuit 230 (e.g., sensor module 176 in FIG. 1), a display 240 (e.g., displaymodule 160 in FIG. 1), a battery 250 (e.g., battery 189 in FIG. 1), and a processor 260(e.g., processor 120 in FIG. 1).
[0046] According to various embodiments of the disclosure, the communicationcircuit 210 (e.g., communication module 190 in FIG. 1) may establish a communicationchannel with an external electronic device (e.g., electronic device 102 in FIG. 1) andsupport transmission and reception of various data to and from the external electronicdevice.
[0047] According to various embodiments of the disclosure, the memory 220(e.g., memory 130 in FIG. 1) may store a program that identifies the gaze direction ofeach of the user's left and right eyes based on light received through one or more lightreceiving pixels constituting the display 240. The memory 220 may store a programthat outputs an image based on the identified gaze directions of the user's left andright eyes. The memory 220 may store a program that adjust the periodicity ofoutputting a video signal and the periodicity of receiving light reflected from the user'sleft and right eyes.
[0048] According to various embodiments of the disclosure, the sensor circuit230 (e.g., sensor module 176 in FIG. 1) may measure a physical quantity or sense anoperating state of the electronic device 201 and convert the measured or sensedinformation into an electrical signal. The sensor circuit 230 may include, for example,at least one of an acceleration sensor, a geomagnetic sensor, a gyro sensor, a magneticsensor, a proximity sensor, or a gesture sensor.
[0049] According to various embodiments of the disclosure, the display 240(e.g., display module 160 in FIG. 1) may include at least one micro light emitting diode(LED). The display 240 may include a first display (not shown) and a second display (notshown). Each of the first display and the second display may include a plurality of pixels(e.g., red pixel, green pixel, blue pixel) for displaying a virtual image, and a lightreceiving pixel disposed between the plurality of pixels to receive light reflected fromthe eye.
[0050] According to various embodiments of the disclosure, one or morebatteries (e.g., battery 189 in FIG. 1) may be provided, and may supply power tocomponents constituting the electronic device 201.
[0051] According to various embodiments of the disclosure, the processor 260(e.g., processor 120 in FIG. 1) may receive light reflected from each of the user's leftand right eyes through one or more light-receiving pixels constituting the display 240(e.g., first display and second display). The processor 260 may identify the gazedirections of the user's left and right eyes based on light received through one or morelight-receiving pixels. For example, the processor 260 may identify the user's gazedirection (e.g., pupil movement) based on light reflected from the user's eyes (e.g.,user's left and right eyes) and output an image based on this.
[0052] According to various embodiments of the disclosure, the processor 260may output an image on a first periodicity. The processor 260 may receive lightreflected from each of the left and right eyes of the user through one or more lightreceiving pixels constituting the display on a second periodicity shorter than (or longerthan, or equal to) the first periodicity and identify the gaze directions of the left andright eyes based on the received light.
[0053] The electronic device 201 according to various embodiments mayinclude a plurality of pixels, light-receiving pixels disposed between the plurality ofpixels, the display 240 composed of the plurality of pixels and the light-receiving pixels,and the processor 260 operatively connected to the display 240, wherein the processor260 may be configured to receive light reflected from each of the user's left and righteyes through one or more light-receiving pixels constituting the display 240, identifythe gaze direction of each of the left and right eyes based on the light received throughthe one or more light-receiving pixels, and output an image based on the identifiedgaze direction.
[0054] According to various embodiments, the processor 260 may be configuredto receive light having been output from the display 240 and reflected from each ofthe user's left and right eyes through the one or more light-receiving pixels, and a paththrough which the light is output and a path through which the light is received throughthe one or more light-receiving pixels may be the same.
[0055] According to various embodiments, the processor 260 may be configuredto adjust the timing of driving the plurality of pixels and the timing of driving the oneor more light-receiving pixels.
[0056] According to various embodiments, the processor 260 may be configuredto: control outputting the image by driving the plurality of pixels on a first periodicity;and receive the light reflected from each of the user's left and right eyes by driving theone or more light-receiving pixels on a second periodicity equal to, or longer than thefirst periodicity.
[0057] According to various embodiments, the processor 260 may be configuredto: control outputting the image by driving the plurality of pixels on a first periodicity;and receive the light reflected from each of the user's left and right eyes by driving theone or more light-receiving pixels on a second periodicity shorter than the firstperiodicity.
[0058] According to various embodiments, the electronic device 201 mayfurther include at least one light source for outputting infrared light, and the processor260 may be configured to receive infrared light having been output through the at leastone light source and reflected from each of the user's left and right eyes through theone or more light-receiving pixels, wherein a path through which the infrared light isoutput and a path through which the infrared light is received through the one or morelight-receiving pixels may be different.
[0059] According to various embodiments, each of the plurality of pixels and theone or more light-receiving pixels may include a micro lens.
[0060] According to various embodiments, the plurality of pixels and the one ormore light-receiving pixels may be arranged to be separated by partition walls.
[0061] According to various embodiments, the display 240 may include signaltransmission paths for controlling the plurality of pixels and signal transmission pathsfor controlling the one or more light-receiving pixels.
[0062] According to various embodiments, if the amount of charge of the oneor more light-receiving pixels having received light reflected from each of the left andright eyes of the user is less than or equal to a specified amount of charge, theprocessor 260 may be configured to receive the light reflected from each of the user'sleft and right eyes by using the one or more light-receiving pixels and one or more lightreceiving pixels arranged close to the one or more light-receiving pixels.
[0063] FIG. 3 is a diagram illustrating an electronic device, for example, ARglasses according to an embodiment of the disclosure.
[0064] Referring to FIG. 3, diagram 300 illustrates an electronic device (e.g.,electronic device 201 in FIG. 2) that may be worn on the user's face and provide animage related to an augmented reality service and / or a virtual reality service to theuser.
[0065] In an embodiment, the electronic device 201 may include a first display305, a second display 310, a screen display section 315, an input optical member 320,a first transparent member 325a, a second transparent member 325b, lighting units330a and 330b, a first PCB 335a, a second PCB 335b, a first hinge 340a, a second hinge340b, a first camera 345, a plurality of microphones (e.g., first microphone 350a,second microphone 350b, third microphone 350c), a plurality of speakers (e.g., firstspeaker 355a, second speaker 355b), a battery 360, a second camera 365a, and a thirdcamera 365b.
[0066] In an embodiment, the display (e.g., first display 305, second display 310)may include, for example, liquid crystal display (LCD), digital mirror device (DMD),liquid crystal on silicon (LCoS), organic light emitting diodes (OLED), or micro lightemitting diodes (micro LED). Although not shown, when the display is implementedwith one of liquid crystal display, digital mirror device, and liquid crystal on silicon, theelectronic device 201 may include a light source for radiating light to a screen outputarea of the display. In another embodiment, when the display is capable of generatinglight by itself, for example, when the display is implemented with either organic lightemitting diodes or micro LEDs, the electronic device 201 may provide a good qualityvirtual image to the user even without a separate light source. In an embodiment,when the display is implemented with organic light emitting diodes or micro LEDs, asa light source is not necessary, the electronic device 201 can be lightened in weight.
[0067] The display (e.g., first display 305, second display 310) according tovarious embodiments of the disclosure may include at least one micro light emittingdiode (LED). For example, a micro LED can express red (R), green (G), and blue (B) byself-emission; as the size is small (e.g., about 100 μm or less), one chip can implementone pixel (e.g., one of R, G and B). Hence, when the display is composed of micro LEDs,high resolution can be provided without a backlight unit (BLU).
[0068] It is not limited thereto, and one pixel may include R, G and B, and onechip may be implemented with a plurality of pixels including R, G and B.
[0069] In an embodiment, the display (e.g., first display 305, second display 310)may include a display area composed of pixels for displaying a virtual image, and lightreceiving pixels (e.g., photo sensor pixels) that are arranged between the pixels andreceive light reflected from eyes and convert it into electrical energy and output it.
[0070] In an embodiment, the electronic device 201 (e.g., processor 260 in FIG.2) may detect the user's gaze direction (e.g., pupil movement) through light-receivingpixels. For example, the electronic device 201 may detect and track a gaze direction forthe user's right eye and a gaze direction for the user's left eye through one or morelight-receiving pixels constituting the first display 305 and one or more light-receivingpixels constituting the second display 310. The electronic device 201 may determinethe location of the center of a virtual image according to gaze directions of the user'sright and left eyes detected through one or more light-receiving pixels (e.g., directionsin which pupils of the user's right and left eyes gaze).
[0071] In an embodiment, light emitted from the display (e.g., first display 305,second display 310) may pass through a lens (not shown) and a waveguide and reachthe screen display section 315 formed on the first transparent member 325a disposedto face the user's right eye and the screen display section 315 formed on the secondtransparent member 325b disposed to face the user's left eye. For example, lightemitted from the display (e.g., first display 305, second display 310) may pass throughthe waveguide and be reflected on the grating area formed on the input opticalmember 320 and the screen display section 315 and be delivered to the user's eyes.The first transparent member 325a and / or the second transparent member 325b maybe made of a glass plate, a plastic plate, or a polymer, and can be made transparent ortranslucent.
[0072] In an embodiment, a lens (not shown) may be disposed in front of thedisplay (e.g., first display 305, second display 310). The lens (not shown) may include aconcave lens and / or a convex lens. For example, the lens (not shown) may include aprojection lens or a collimation lens.
[0073] In an embodiment, the screen display section 315 or the transparentmember (e.g., first transparent member 325a, second transparent member 325b) mayinclude a lens including a waveguide, or a reflective lens.
[0074] In an embodiment, the waveguide may be made of glass, plastic, orpolymer, and may include a nano pattern, for example, a polygonal or curved gratingstructure, formed on an inner or outer surface. According to an embodiment, lightincident to one end of the waveguide may be propagated inside the display waveguideby the nano pattern and delivered to the user. In an embodiment, a waveguidecomposed of a freeform prism may provide incident light to the user through areflection mirror. The waveguide may include at least one of at least one diffractiveelement (e.g., diffractive optical element (DOE), holographic optical element (HOE)) ora reflective element (e.g., reflective mirror). In an embodiment, the waveguide mayguide light emitted from the display (e.g., first display 305 or second display 310) tothe user's eyes by using at least one diffractive element or a reflective elementincluded in the waveguide.
[0075] According to various embodiments, the diffractive element may includean input optical member 320 and an output optical member (not shown). For example,the input optical member 320 may mean an input grating area, and the output opticalmember (not shown) may mean an output grating area. The input grating area mayserve as an input end for diffracting (or reflecting) light output from the display (e.g.,first display 305, second display 310) (e.g., micro LEDs) to transfer the light to thetransparent member (e.g., first transparent member 325a, second transparentmember 325b) of the screen display section 315. The output grating area may serve asan outlet for diffracting (or reflecting) light delivered to the transparent member (e.g.,first transparent member 325a, second transparent member 325b) of the waveguideto the user's eyes.
[0076] According to various embodiments, the reflective element may include atotal reflection optical element or a total reflection waveguide for total internalreflection (TIR). For example, total internal reflection, as one way of inducing light, maymean making an incident angle such that light (e.g., virtual image) input through theinput grating area is reflected 100 percent on one surface (e.g., specific surface) of thewaveguide and is delivered 100 percent to the output grating area.
[0077] In an embodiment, the path of light emitted from the display (e.g., firstdisplay 305 or second display 310) may be guided to the waveguide through the inputoptical member 320. Light moving inside the waveguide may be guided toward theuser's eye through the output optical member. The screen display section 315 may bedetermined based on light emitted in a direction of the eye.
[0078] In an embodiment, the first camera 345 may be referred to as highresolution (HR) or photo video (PV), and may include a high resolution camera. Thefirst camera 345 may include a color camera equipped with functions for obtaininghigh-quality images such as auto focus (AF) and optical image stabilization (OIS).Without being limited thereto, the first camera 345 may include a global shutter (GS)camera or a rolling shutter (RS) camera.
[0079] In an embodiment, the second camera 365a and the third camera 365bmay include a camera used for 3 degrees of freedom (3DoF) or 6 degrees of freedom(6DoF) head tracking, hand detection and tracking, and gesture and / or spacerecognition. For example, the second camera 365a and the third camera 365b mayinclude a global shutter (GS) camera to detect and track the movement of the headand hand.
[0080] In an embodiment, at least one sensor (e.g., sensor circuit 230 in FIG. 2)(e.g., gyro sensor, accelerometer sensor, geomagnetic sensor, and / or gesture sensor),the second camera 365a, and the third camera 365b may perform at least one of 6DoFhead tracking, pose estimation and prediction, gesture and / or space recognition, orsimultaneous localization and mapping (SLAM) through depth shooting.
[0081] In another embodiment, the second camera 365a and the third camera365b may be used separately as a camera for head tracking and a camera for handtracking.
[0082] In an embodiment, the lighting units 330a and 330b may have differentuses depending on where they are attached. For example, the lighting unit 330a or330b may be attached together with the second camera 365a or the third camera 365bmounted around the hinge (e.g., first hinge 340a, second hinge 340b) connecting theframe and the temple or around the bridge connecting the frames. In the case ofshooting with a GS camera, the lighting units 330a and 330b may be used tosupplement ambient brightness. For example, the lighting units 330a and 330b may beused when it is not easy to detect a subject to be photographed in a dark environmentor due to mixed and reflected light from various light sources.
[0083] In an embodiment, components constituting the electronic device 201(e.g., processor 260 and memory 220 in FIG. 2) may be located on the PCB (e.g., firstPCB 335a, second PCB 335b). The PCB may deliver electrical signals to the componentsconstituting the electronic device 201.
[0084] In an embodiment, the plurality of microphones (e.g., first microphone350a, second microphone 350b, third microphone 350c) may process external soundsignals into electrical voice data. The processed voice data may be utilized in variousways according to the function being performed (or, application being executed) onthe electronic device 201.
[0085] In an embodiment, the plurality of speakers (e.g., first speaker 355a,second speaker 355b) may output audio data received through the communicationcircuit (e.g., communication circuit 210 in FIG. 2) or stored in the memory (e.g.,memory 220 in FIG. 2).
[0086] In an embodiment, one or more batteries (e.g., battery 360) may beincluded, and may supply power to the components constituting the electronic device201.
[0087] FIG. 4A is a diagram illustrating a plan view of pixels constituting a displayaccording to an embodiment of the disclosure.
[0088] Referring to FIG. 4A, diagram 400 illustrates a display (e.g., first display305 or second display 310 in FIG. 3) of the electronic device (e.g., electronic device 201in FIG. 2) may be composed of a plurality of pixels 405, 410 and 415 and a plurality oflight-receiving pixels 420.
[0089] In an embodiment, the plurality of pixels may include a red (R) pixel 405,a green (G) pixel 410, and a blue (B) pixel 415. The electronic device 201 may display avirtual image by using the plurality of pixels 405, 410 and 415.
[0090] In an embodiment, a light-receiving pixel 420 may be disposed between(or on a periphery (e.g., border) of) the plurality of pixels 405, 410 and 415. The lightreceiving pixel 420 disposed between (or on a periphery (e.g., border) of) the pluralityof pixels 405, 410 and 415 may receive light reflected from the user's eye. Theelectronic device 201 may detect the user's gaze direction (e.g., pupil movement)based on reflected light received by light-receiving pixels 420. For example, lightemitted from the first display 305 may be reflected by the user's right eye and receivedby light-receiving pixels 420 constituting the first display 305. Light emitted from thesecond display 310 may be reflected by the user's left eye and received by lightreceiving pixels 420 constituting the second display 310. The electronic device 201 maydetect and track a gaze direction for the user's right eye and a gaze direction for theuser's left eye based on light received by light-receiving pixels 420 constituting the firstdisplay 305 and the second display 310. The electronic device 201 may determine thelocation of the center of a virtual image based on the detected gaze directions of theuser's right and left eyes.
[0091] FIG. 4B is a diagram illustrating an arrangement structure of pixelsconstituting a display according to an embodiment of the disclosure.
[0092] Referring to FIG. 4B, diagram 425 illustrates a plurality of pixels 405, 410and 415 and the light-receiving pixel 420 being arranged in various forms. For example,the plurality of pixels 405, 410 and 415 and the light-receiving pixel 420 may bearranged in a pentile structure.
[0093] In an embodiment, when the plurality of pixels 405, 410 and 415 and thelight-receiving pixel 420 constituting the display (e.g., first display 305 or second display310) are arranged in a pentile structure, as indicated by indicia <430>, one red (R) pixel405, two green (G) pixels 410, and one blue (B) pixel 415, whose shape is square, maybe arranged, and a rectangular light-receiving pixel 420 may be disposed between thered (R) pixel 405, the two green (G) pixels 410, and the blue (B) pixel 415. Therectangular light-receiving pixels 420 may have different sizes or shapes.
[0094] The arrangement structure of the plurality of pixels 405, 410 and 415 andthe light-receiving pixel 420 according to various embodiments is not limited to apentile structure. For example, the plurality of pixels 405, 410 and 415 and the lightreceiving pixel 420 may be arranged in various structures such as an RGB structure ora stripe structure.
[0095] According to various embodiments, to obtain accurate information aboutthe user's gaze by increasing light reception efficiency in a dark environment wherelight is insufficient, among the plurality of pixels 405, 410 and 415 and the lightreceiving pixel 420, the area of the light-receiving pixel 420 may be widened (e.g., thearea occupied by the light-receiving pixel 420 may be increased).
[0096] For example, as indicated by indicia <435>, one red (R) pixel 405, twogreen (G) pixels 410, and one blue (B) pixel 415, whose shape is octahedral, may bedisposed, and a rhombic light-receiving pixel 420 may be disposed between the red (R)pixel 405, the two green (G) pixels 410, and the blue (B) pixel 415.
[0097] As another example, as indicated by indicia <440>, one red (R) pixel 405,two green (G) pixels 410, and one blue (B) pixel 415, whose shape is rhombic, may bedisposed, and a rhombic light-receiving pixel 420 may be disposed between the red (R)pixel 405, the two green (G) pixels 410, and the blue (B) pixel 415.
[0098] According to various embodiments, the area occupied by the lightreceiving pixel 420 indicated by indicia <440> may be larger than that of the lightreceiving pixel 420 indicated by indicia <435>. As the area occupied by the lightreceiving pixel 420 indicated by indicia <440> is larger than the area occupied by thelight-receiving pixel 420 indicated by indicia <435>, the light-receiving efficiency of thelight-receiving pixel 420 indicated by indicia <440> may be higher than that of the lightreceiving pixel 420 indicated by indicia <435>.
[0099] It has been described in FIG. 4B according to various embodiments thatthe area occupied by the light-receiving pixel 420 is increased to heighten the lightreceiving efficiency, without being limited thereto.
[00100] In an embodiment, the electronic device 201 may check whether thecharge amount of one or more light-receiving pixels receiving light reflected from eachof the user's left and right eyes is less than or equal to a specified charge amount. Ifthe charge amount of one or more light-receiving pixels receiving light reflected fromeach of the user's left and right eyes is less than or equal to the specified chargeamount, the electronic device 201 may receive light reflected from each of the user'sleft and right eyes by using one or more light-receiving pixels and one or more lightreceiving pixels arranged close to the one or more light-receiving pixels. Light reflectedfrom the user's right and left eyes may be converted into electrical signals by one ormore light-receiving pixels and one or more light-receiving pixels arranged close to theone or more light-receiving pixels. The converted electrical signals may be transmittedto a processor, for example, an image signal processor (ISP) (not shown). The imagesignal processor may perform image processing in a manner suitable for tracking theuser's right and left eyes, and may, based on this, identify the location of the user'spupils and track the direction in which the pupils move, for example, the gaze direction.
[00101] According to various embodiments, if the charge amount of one or morelight-receiving pixels receiving light reflected from each of the user's left and right eyesis less than or equal to the specified charge amount, the electronic device 201 mayreceive light reflected from each of the user's right and left eyes by using one or morelight-receiving pixels and one or more light-receiving pixels arranged close to the oneor more light-receiving pixels. Accordingly, the light receiving efficiency may beincreased. In other words, the electronic device 201 may utilize one or more lightreceiving pixels and one or more light-receiving pixels arranged close to the one ormore light-receiving pixels to receive light reflected from each of the user's right andleft eyes, and thus the user's gaze information can be accurately obtained.
[00102] According to various embodiments, the plurality of pixels 405, 410 and415 and the light-receiving pixel 420 may be arranged in a configuration in which atleast two of the structures indicated above by indicia <430>, <435>, and <440> aremerged.
[00103] FIG. 4C is a view illustrating a cross-sectional view of a plurality of pixelsconstituting a display according to an embodiment of the disclosure.
[00104] Referring to FIG. 4C, view 450 illustrates a display (e.g., first display 305,second display 310 in FIG. 3) being composed of at least one micro light emitting diode(micro LED). The display may include a blue (B) pixel 415, a red (R) pixel 405, a green(G) pixel 410, and a light-receiving pixel 420.
[00105] In an embodiment, each of the blue (B) pixel 415, red (R) pixel 405, andgreen (G) pixel 410 may include a color filter 453 and an LED 451 for causing the pixelto emit light. The color filter 453 may include a blue filter, a red filter, and a green filter.Without being limited thereto, the color filter 453 may include a yellow filter, amagenta filter, or a cyan filter.
[00106] In various embodiments, each of the blue (B) pixel 415, the red (R) pixel405, and the green (G) pixel 410 has been described as including the color filter 453,but it is not limited thereto. For example, each of the blue (B) pixel 415, the red (R)pixel 405, and the green (G) pixel 410 may include a no color filter (not shown) and theLED 451 for causing the pixel to emit light.
[00107] In an embodiment, the light-receiving pixel 420 may include a no colorfilter 463.
[00108] In an embodiment, to guide light in a specific direction, each of the blue(B) pixel 415, the red (R) pixel 405, the green (G) pixel 410, and the light-receiving pixel420 may include a micro lens 470. For example, the micro lens 470 may be formed onthe color filter 453 in correspondence to each of the blue (B) pixel 415, the red (R) pixel405, the green (G) pixel 410, and the light-receiving pixel 420.
[00109] In an embodiment, to prevent the light passing through the micro lens470 from being reflected or mixed with neighboring pixels due to unintended causes,each of the blue (B) pixel 415, the red (R) pixel 405, the green (G) pixel 410, and thelight-receiving pixel 420 may be separated by a partition wall 480. As each pixel (e.g.,blue (B) pixel 415, red (R) pixel 405, green (G) pixel 410, and light-receiving pixel 420)is separated by the partition wall 480, visibility can be improved by preventing theemitted light from being mixed.
[00110] In various embodiments, the shape of the partition wall is not limited tothe shape shown in FIG. 4C. For example, the partition wall may be formed in variousshapes according to the shape and size of each pixel.
[00111] FIGS. 5A and 5B are diagrams for depicting a path of visible light outputfrom a display and a path of visible light reflected from a user's eye according to variousembodiments of the disclosure.
[00112] Referring to FIG. 5A, according to an embodiment, diagram 500Aillustrates a situation in which light emitted from a display 520 (e.g., first display 305,second display 310 in FIG. 3) is assumed to be visible light (e.g., RGB visible light). Thedisplay 520 (e.g., first display 305, second display 310) may be composed of a pluralityof pixels (e.g., red pixel 405, green pixel 410, blue pixel 415 in FIG. 4A) and a lightreceiving pixel (e.g., light-receiving pixel 420 in FIG. 4A) disposed between the pluralityof pixels 405, 410 and 415.
[00113] In an embodiment, a lens 530 may be disposed in front of the display520. Visible light emitted from the display 520 (e.g., first display 305, second display310) may pass through the lens 530 (e.g., projection lens or collimation lens). The lens530 may include a concave lens and / or a convex lens.
[00114] In an embodiment, among the RGB visible light emitted from the display520, blue (B) visible light passing through the lens 530 may reach a user's eye 510through a first layer 540a of the waveguide. For example, blue (B) visible light emittedfrom the display 520 may be reflected by an input grating 550 of the first layer 540a ofthe waveguide, and the reflected blue (B) visible light may be totally reflected to reachan output grating 560 and be reflected by the output grating 560 to reach the user'seye 510. Among the RGB visible light emitted from the display 520, green (G) visiblelight passing through the lens 530 may reach the user's eye 510 through a second layer540b of the waveguide. For example, green (G) visible light emitted from the display520 may be reflected by the input grating 550 of the second layer 540b of thewaveguide, and the reflected green (G) visible light may be totally reflected to reachthe output grating 560 and be reflected by the output grating 560 to reach the user'seye 510. Among the RGB visible light emitted from the display 520, red (R) visible lightpassing through the lens 530 may reach the user's eye 510 through a third layer 540cof the waveguide. For example, red (R) visible light emitted from the display 520 maybe reflected by the input grating 550 of the third layer 540c of the waveguide, and thereflected red (R) visible light may be totally reflected to reach the output grating 560and be reflected by the output grating 560 to reach the user's eye 510.
[00115] Referring to FIG. 5B, according to an embodiment, diagram 500Billustrates visible light (e.g., RGB visible light) reaching the user's eye 510 throughindividual layers (e.g., first layer 540a, second layer 540b, third layer 540c) of thewaveguide being reflected from the user's eye 510. The light-receiving pixel 420 mayreceive the reflected visible light through individual layers (e.g., first layer 540a, secondlayer 540b, third layer 540c) of the waveguide and the lens 530.
[00116] According to various embodiments, when the light output from thedisplay 520 (e.g., first display 305, second display 310) is visible light, the path throughwhich the visible light reaches the user's eye 510 and the path through which thevisible light reflected from the user's eye 510 is received may be the same. Forexample, blue (B) visible light reaching the user's eye 510 through the first layer 540aof the waveguide may be reflected from the user's eye 510 and received by the lightreceiving pixel 420 through the first layer 540a. Green (G) visible light reaching theuser's eye 510 through the second layer 540b of the waveguide may be reflected fromthe user's eye 510 and received by the light-receiving pixel 420 through the secondlayer 540b. Red (R) visible light reaching the user's eye 510 through the third layer 540cof the waveguide may be reflected from the user's eye 510 and received by the lightreceiving pixel 420 through the third layer 540c.
[00117] In FIGS. 5A and 5B according to various embodiments, the waveguide isshown as being composed of three layers, for example, first layer 540a, second layer540b, and third layer 540c, but it is not limited thereto. For example, the waveguidemay be composed of two layers or one layer.
[00118] FIG. 6 is a diagram for depicting paths of visible light and infrared lightoutput from a display according to an embodiment of the disclosure.
[00119] Referring to FIG. 6, according to an embodiment, diagram 600 illustratesvisible light (e.g., RGB visible light) being emitted from a display 620 (e.g., first display305, second display 310 in FIG. 3). One or more infrared LEDs (not shown) for emittinginfrared rays may be disposed in the frame of the electronic device (e.g., electronicdevice 201 in FIG. 2).
[00120] In an embodiment, RGB visible light may be emitted from the display 620(e.g., the first display 305 and the second display 310), and infrared light may beemitted from the one or more infrared LEDs.
[00121] In an embodiment, the display 620 (e.g., first display 305, second display310) may be composed of a plurality of pixels (e.g., red pixel 405, green pixel 410, bluepixel 415 in FIG. 4A) and a light-receiving pixel (e.g., light-receiving pixel 420 in FIG. 4A)disposed between the plurality of pixels 405, 410 and 415.
[00122] In an embodiment, a lens 630 may be disposed in front of the display620. Visible light emitted from the display 620 (e.g., first display 305, second display310) may pass through the lens 630 (e.g., projection lens or collimation lens).
[00123] In an embodiment, among the RGB visible light emitted from the display620, blue (B) visible light passing through the lens 630 may reach a user's eye 610through a first layer 640a of the waveguide. Among the RGB visible light emitted fromthe display 620, green (G) visible light passing through the lens 630 may reach theuser's eye 610 through a second layer 640b of the waveguide. Among the RGB visiblelight emitted from the display 620, red (R) visible light passing through the lens 630may reach the user's eye 610 through a third layer 640c of the waveguide.
[00124] In an embodiment, infrared light may be emitted from one or moreinfrared LEDs disposed in the frame of the electronic device 201, and the emittedinfrared light may be delivered to the user's eye 610. The light-receiving pixel 420 mayreceive reflected infrared light through a fourth layer 640d (e.g., infrared (IR) layer) ofthe waveguide and the lens 630.
[00125] In FIG. 6, according to various embodiments, when visible light is outputfrom the display 620 (e.g., first display 305 or second display 310) and infrared light isoutput from the at least one infrared LED, the path through which visible light reachesthe user's eye 510 and the path through which infrared light reflected from the user'seye 610 is received may be different. For example, whereas RGB visible light may reachthe user's eye 610 through several layers (e.g., first layer 640a, second layer 640b, thirdlayer 640c) of the waveguide, infrared light reflected from the user's eye 610 may bereceived by the light-receiving pixel 420 through the fourth layer 640d.
[00126] Without being limited thereto, in various embodiments, during a period(time) in which visible light is not output from the display 620 (e.g., first display 305,second display 310), infrared light may be received by the light-receiving pixel 420through the first to third layers (e.g., layers 640a, 640b, 640c) as well as the fourth layer640d of the waveguide.
[00127] In FIG. 6, according to various embodiments, the waveguide is shown asbeing composed of four layers, for example, first layer 640a, second layer 640b, thirdlayer 640c, and fourth layer 640d, but it is not limited thereto. For example, thewaveguide may be composed of two layers or one layer.
[00128] FIG. 7 is a block diagram illustrating a display according to anembodiment of the disclosure.
[00129] Referring to FIG. 7, block diagram 700 illustrates a display (e.g., firstdisplay 305, second display 310 in FIG. 3) may include a display panel 710 and a displaydriver IC (DDI) for display control. The DDI may include a gate driver 720, a data driver730, a timing controller 740, and an interface block 750.
[00130] In an embodiment, the display (e.g., first display 305 or second display310) may include a plurality of gate lines GL1 to GLn, a plurality of data lines DL1 toDLm crossing them. The display panel 710 may include a plurality of pixels (P) formedin regions where the gate lines GL and the data lines DL cross one another.
[00131] In an embodiment, the display (e.g., first display 305 or second display310) may further include a plurality of light-receiving pixel lines PDL1 to PDLk. A lightreceiving pixel PD (e.g., light-receiving pixel 420 in FIG. 4A) may be disposed betweena plurality of sub-pixels (e.g., red (R) pixel 405, green (G) pixel 410, blue (B) pixel 415 inFIG. 4A). The light-receiving pixel lines PDL may be connected to the light-receivingpixels PD (e.g., light-receiving pixel 420). The light-receiving pixel PD may include alight-receiving photodiode.
[00132] In an embodiment, each pixel P may include a plurality of sub-pixels (e.g.,red (R) pixel 405, green (G) pixel 410, blue (B) pixel 415). Each sub-pixel may be formedto include a light emitting diode (e.g., micrometer-sized light emitting diode (microLED)) to emit light of the corresponding color, and a driving circuit (not shown) fordriving the light emitting diode. The driving circuit may be configured to include at leastone thin film transistor and at least one capacitor, and it may be electrically connectedto one of the gate lines GL and may be electrically connected to one of the data linesDL. The driving circuit may charge the capacitor with a data voltage supplied throughthe connected data line DL from the data driver 222 in response to a scan signalreceived through the connected gate line GL from the gate driver 221. The drivingcircuit may control the amount of current supplied to the LED according to the datavoltage charged in the capacitor. That is, individual sub-pixels may display visualinformation based at least on the scan signal and the data signal.
[00133] Although not shown, the display (e.g., first display 305 or second display310) may be configured to further include, in addition to the signal lines, various signallines (e.g., Sens lines, L-TEST lines, TEST lines) for driving the pixels P, and a plurality ofpower lines (e.g., VDD lines, VSS lines, Vcas lines) for supplying power to the pixels (P).
[00134] In an embodiment, the gate driver 720 may supply scan signals to theplurality of gate lines GL1 to GLn according to at least one gate control signal GCSprovided from the timing controller 740. The gate driver 720 may include a gate shiftregister to output a scan signal or scan pulse. The scan signal is sequentially suppliedto individual pixels P, and may be composed of a single signal or multiple signals. Whenthe scan signal is composed of multiple signals, each gate line GL may be composed ofa plurality of lines to supply multiple scan signals to each pixel P.
[00135] In an embodiment, the data driver 730 may convert image data RGBprovided from the timing controller 740 into a data voltage according to at least onedata control signal DCS provided from the timing controller 740. The data driver 730may generate a data voltage by using multiple gamma correction voltages. The datadriver 730 may sequentially supply the generated data voltage to a plurality of pixels Pline by line, for example, in a row-wise manner. The data driver 730 may include a datashift register that outputs a sampling signal, a latch circuit that latches image data rowby row in response to a sampling signal, and a digital analog converter that convertslatched image data into an analog grayscale voltage (pixel voltage).
[00136] In an embodiment, a light-receiving driver 760 may deliver power,control signals, and information to the light-receiving pixels (PD) (e.g., light-receivingpixel 420) through a plurality of light-receiving pixel lines PDL1 to PDLk. Each lightreceiving pixel (PD) may receive light reflected from the user's eye under the controlof the light-receiving driver 760.
[00137] In an embodiment, when the brightness of light reflected from each ofthe user's eyes, for example, the left and right eyes, is less than or equal to a specifiedbrightness value, the light-receiving driver 760 may deliver power, control signals, andinformation to one or more corresponding light-receiving pixels through the lightreceiving pixel lines PDL1 to PDLk so as to receive light reflected from each of the user'sleft and right eyes by using one or more light-receiving pixels disposed close to the oneor more light-receiving pixels PD.
[00138] In an embodiment, the timing controller 740 may align video data RGBprovided from the interface block 750 according to the size and resolution of thedisplay. The timing controller 740 may supply aligned video data RGB to the data driver730. The timing controller 740 may transmit a plurality of control signals GCS and DCSby using at least one synchronization signal SYNC provided from the interface block750. The plurality of control signals GCS and DCS may include at least one gate controlsignal GCS and at least one data control signal DCS. The gate control signal GCS may bea signal for controlling the driving timing of the gate driver 720. The data control signalDCS may be a signal for controlling the driving timing of the data driver 730. Thesynchronization signals SYNC may include a dot clock (DCLK), a data enable signal (DE),a horizontal synchronization signal (Hsync), or a vertical synchronization signal (Vsync).
[00139] In an embodiment, the interface block 750 may receive video data RGBfrom the processor (e.g., processor 120 in FIG. 1), for example, an applicationprocessor, and transmit the received video data RGB to the timing controller 740. Theinterface block 750 may generate at least one synchronization signal SYNC and transmitit to the timing controller 740. The interface block 750 may control a power supply 770to supply at least one driving voltage to the display.
[00140] In an embodiment, the power supply 770 may generate at least onedriving voltage necessary for driving the display and supply the generated drivingvoltage to the display. The at least one driving voltage may include, for example, VDD,VSS (GND), gate-on voltage, gate-off voltage, or initialization voltage. The gate-onvoltage may be a voltage for turning on at least one thin film transistor included in thedisplay. The gate-off voltage may be a voltage for turning off at least one thin filmtransistor included in the display. The initialization voltage may be a voltage forinitializing a specific node in the driving circuit for driving at least one sub-pixel amonga plurality of sub-pixels.
[00141] FIG. 8 is a flowchart illustrating a method of detecting a user's gazedirection and outputting an image based on the detected gaze direction according toan embodiment of the disclosure.
[00142] It may be understood that operations 810, 820, and 830 of FIG. 8according to various embodiments are performed by the processor 260 of theelectronic device (e.g., electronic device 201 in FIG. 2).
[00143] Referring to FIG. 8, flowchart 800 illustrates, at operation 810, theelectronic device 201 receiving light reflected from each of the user's left eye and righteye through one or more light-receiving pixels (e.g., light-receiving pixel 420 in FIG. 4A)constituting the display (e.g., first display 305, second display 310 in FIG. 3).
[00144] In an embodiment, the display (e.g., first display 305 or second display310) may be composed of at least one micro light emitting diode (micro LED). Thedisplay may include the first display 305 and the second display 310. Each of the firstdisplay 305 and the second display 310 may include a plurality of pixels (e.g., red pixel405, green pixel 410, blue pixel 415 in FIG. 4A) for displaying a virtual image, and alight-receiving pixel 420 disposed between the plurality of pixels to receive lightreflected from the eye.
[00145] In an embodiment, light emitted from the first display 305 and thesecond display 310 may pass through a lens (e.g., lens 530 in FIGS. 5A and 5B, lens 630in FIG. 6) (e.g., projection lens or collimation lens). Light passing through the lens 530may reach the user's eyes (e.g., user's left and right eyes) through a waveguide (e.g.,waveguide (e.g., layers 540a, 540b, and 540c) in FIGS. 5A and 5B, waveguide (e.g.,layers 640a, 640b, 640c, and 640d) in FIG. 6). Light reaching the user's eyes (e.g., user'sleft and right eyes) may be reflected from the user's eyes (e.g., user's left and righteyes). The light-receiving pixel 420 may receive light reflected from the user's eyes(e.g., user's left and right eyes). For example, light emitted from the first display 305may be reflected from the user's right eye and received by one or more light-receivingpixels constituting the first display 305. Light emitted from the second display 310 maybe reflected from the user's left eye and received by one or more light-receiving pixelsconstituting the second display 310.
[00146] In an embodiment, at operation 820, the electronic device 201 mayidentify the gaze directions of the user's left and right eyes based on the light receivedthrough one or more light-receiving pixels.
[00147] In an embodiment, the electronic device 201 may detect the user's gazedirection (e.g., pupil movement) based on light reflected from the user's eyes (e.g.,user's left and right eyes). For example, the electronic device 201 may detect and trackthe gaze direction of the user's right eye and the gaze direction of the user's left eyebased on light received by one or more light-receiving pixels provided in the firstdisplay 305 and the second display 310. Light reflected from the user's right and lefteyes may be converted into electrical signals at one or more light-receiving pixelsprovided in the first display 305 and the second display 310. The converted electricalsignals may be transmitted to a processor, for example, an image signal processor (ISP)(not shown). The image signal processor may perform image processing in a mannersuitable for tracking the user's right and left eyes, and may, based on this, identify thelocation of the user's pupils and track the direction in which the pupils move, forexample, the gaze direction.
[00148] In an embodiment, at operation 830, the electronic device 201 mayoutput an image based on the identified gaze direction.
[00149] In an embodiment, the electronic device 201 may determine the locationof the screen display section (e.g., screen display section 315 in FIG. 3) to output avirtual image based on the direction in which the tracked user's eyes move. Forexample, the position of the screen display section 315 on the first transparentmember (e.g., first transparent member 325a in FIG. 3) disposed to face the user's righteye and the second transparent member (e.g., second transparent member 325b inFIG. 3) disposed to face the user's left eye may be determined. In other words, theelectronic device 201 may determine the location of the center of a virtual imageaccording to gaze directions of the user's right and left eyes (e.g., directions in whichthe pupils of the user's right and left eyes gaze).
[00150] In an embodiment, the electronic device 201 may output an image to thescreen display section 315.
[00151] FIG. 9 is a flowchart illustrating a method for an electronic device tooutput an image by adjusting the periodicity of outputting a video signal and theperiodicity of receiving reflected light according to an embodiment of the disclosure.
[00152] It may be understood that operations 910 and 920 of FIG. 9 according tovarious embodiments are performed by the processor 260 of the electronic device(e.g., electronic device 201 in FIG. 2).
[00153] Referring to FIG. 9, flowchart 900 illustrates, at operation 910, theelectronic device 201 outputting an image on a first periodicity. For example, theelectronic device 201 may control the display (e.g., first display 305, second display 310in FIG. 3) to output a virtual image on a specific periodicity (e.g., 120Hz, 240Hz).
[00154] In an embodiment, the electronic device 201 may set the timing fordriving pixels (e.g., red pixel 405, green pixel 410, blue pixel 415 in FIG. 4A) to outputa virtual image and the timing for driving light-receiving pixels (e.g., light-receiving pixel420 in FIG. 4A) to receive light reflected from the user's eyes (e.g., user's right and lefteyes).
[00155] In an embodiment, at operation 920, the electronic device 201 mayreceive light reflected from each of the user's left and right eyes through one or morelight-receiving pixels constituting the display (e.g., first display 305 or second display310) on a second periodicity shorter than the first periodicity, and identify the gazedirections of the left and right eyes based on the received light.
[00156] Although the second periodicity has been described as being shorterthan the first periodicity according to various embodiments, it is not limited thereto.For example, the second periodicity may be longer than, or equal to the firstperiodicity.
[00157] Various embodiments will be described with reference to FIG. 10 belowin relation to adjusting the periodicity of outputting a video signal and the periodicityof receiving the reflected light.
[00158] A method of obtaining user gaze information by using light-receivingpixels 420 of the display (e.g., first display 305 or second display 310) of the electronicdevice 201 according to various embodiments may include: receiving light reflectedfrom each of the user's left and right eyes through one or more light-receiving pixels420 constituting the display (e.g., first display 305 or second display 310); identifyingthe gaze direction of each of the left and right eyes based on the light received throughthe one or more light-receiving pixels 420; and outputting an image based on theidentified gaze direction.
[00159] According to various embodiments, receiving the light reflected fromeach of the user's left and right eyes through the one or more light-receiving pixels 420may include receiving light having been output from the display (e.g., first display 305or second display 310) and reflected from each of the user's left and right eyes throughthe one or more light-receiving pixels 420, wherein the path through which the light isoutput and the path through which the light is received via the one or more lightreceiving pixels 420 may be the same.
[00160] The method of obtaining user gaze information by using light-receivingpixels 420 of the display (e.g., first display 305 or second display 310) of the electronicdevice 201 according to various embodiments may further include adjusting the timingfor driving the plurality of pixels and the timing for driving the one or more lightreceiving pixels 420.
[00161] According to various embodiments, outputting the image may includeoutputting an image by driving the plurality of pixels 405, 410 and 415 on a firstperiodicity; the method of obtaining user gaze information by using light-receivingpixels 420 of the display (e.g., first display 305 or second display 310) of the electronicdevice 201 may further include receiving the light reflected from each of the user's leftand right eyes by driving the one or more light-receiving pixels 420 on a secondperiodicity equal to, or longer than the first periodicity.
[00162] According to various embodiments, outputting the image may includeoutputting an image by driving the plurality of pixels 405, 410 and 415 on a firstperiodicity; the method of obtaining user gaze information by using light-receivingpixels 420 of the display (e.g., first display 305, second display 310) of the electronicdevice 201 may further include receiving the light reflected from each of the user's leftand right eyes by driving the one or more light-receiving pixels 420 on a secondperiodicity shorter than the first periodicity.
[00163] According to various embodiments, receiving the light reflected fromeach of the user's left and right eyes through one or more light-receiving pixels 420may include receiving infrared light having been output from at least one light sourceand reflected from each of the user's left and right eyes through the one or more lightreceiving pixels 420, wherein the path through which the infrared light is output andthe path through which the infrared light is received via the one or more light-receivingpixels 420 may be different.
[00164] According to various embodiments, each of the plurality of pixels 405,410 and 415 and the one or more light-receiving pixels 420 may include a micro lens.
[00165] According to various embodiments, the plurality of pixels 405, 410 and415 and the one or more light-receiving pixels 420 may be arranged to be separatedfrom each other by partition walls.
[00166] According to various embodiments, the display (e.g., first display 305 orsecond display 310) may include signal transmission paths for controlling the pluralityof pixels 405, 410 and 415, and signal transmission paths for controlling the one ormore light-receiving pixels 420.
[00167] The method of obtaining user gaze information by using light-receivingpixels 420 of the display (e.g., first display 305, second display 310) of the electronicdevice 201 may further include: determining, after receiving light reflected from eachof the user's left and right eyes through one or more light-receiving pixels 420constituting the display (e.g., first display 305 or second display 310), whether thecharge amount of the one or more light-receiving pixels 420 is less than or equal to aspecified charge amount; and receiving, upon determining that the charge amount ofthe one or more light-receiving pixels 420 is less than the specified charge amount,light reflected from each of the user's left and right eyes by using the one or more lightreceiving pixels 420 and one or more light-receiving pixels 420 arranged close to theone or more light-receiving pixels 420.
[00168] FIG. 10 is a diagram for depicting the periodicity of outputting a videosignal and the periodicity of receiving reflected light in an electronic device accordingto an embodiment of the disclosure.
[00169] According to various embodiments, a display 1030 (e.g., first display 305,second display 310 in FIG. 3) of the electronic device (e.g., electronic device 201 in FIG.2) may be composed of pixels (e.g., red (R) pixel 405, green (G) pixel 410, blue (B) pixel415 in FIG. 4A) for outputting a virtual image, and light-receiving pixels (e.g., lightreceiving pixel 420 in FIG. 4A) that are arranged between the pixels and receive lightreflected from a user's eye 1010 (e.g., user's right eye and / or left eye), convert thereceived light into electric energy, and output it.
[00170] Referring to FIG. 10, diagram 1000 illustrates the display 1030 beingdriven at a specific frequency 1020 (e.g., about 120Hz or about 240Hz) to output avirtual image.
[00171] In an embodiment, the electronic device 201 may set a timing for drivingthe plurality of pixels 405, 410 and 415 to output a virtual image and a timing fordriving the light-receiving pixels 420 to receive light reflected from the user's eye 1010(e.g., user's right eye and / or left eye).
[00172] In an embodiment, the operation of driving the plurality of pixels 405,410 and 415 to output a virtual image and the operation of driving the light-receivingpixels 420 to receive light reflected from the user's dhanalakshmi@bhel.in1010 may beswitched at a specific frame rate (e.g., about 0.5 frames per second (fps) or about 0.25fps). For example, the electronic device 201 may output a virtual image (1045) bydriving the pixels (e.g., red (R) pixel 405, green (G) pixel 410, blue (B) pixel 415) foroutputting a virtual image during a half frame period 1050 of one frame period 1040,and may detect an output optical signal having been reflected from the user's eye 1010(e.g., user's right and left eyes) (1055) by driving the light-receiving pixels 420 duringthe remaining half frame period 1060.
[00173] According to various embodiments, as described above, the timing fordriving the plurality of pixels 405, 410 and 415 to output a virtual image and the timingfor driving the light-receiving pixels 420 to receive light reflected from the user's eye1010 (e.g., user's right and left eyes) may be set differently, thereby reducinginterference between pixels.
[00174] The electronic device according to various embodiments may be one ofvarious types of electronic devices. The electronic devices may include, for example, aportable communication device (e.g., a smartphone), a computer device, a portablemultimedia device, a portable medical device, a camera, a wearable device, or a homeappliance. According to an embodiment of the disclosure, the electronic devices arenot limited to those described above.
[00175] It should be appreciated that various embodiments of the disclosure andthe terms used therein are not intended to limit the technological features set forthherein to particular embodiments and include various changes, equivalents, orreplacements for a corresponding embodiment. With regard to the description of thedrawings, similar reference numerals may be used to refer to similar or relatedelements. As used herein, each of such phrases as "A or B," "at least one of A and B,""at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one ofA, B, or C," may include any one of, or all possible combinations of the itemsenumerated together in a corresponding one of the phrases. As used herein, suchterms as "1st" and "2nd," or "first" and "second" may be used to simply distinguish acorresponding component from another, and does not limit the components in otheraspect (e.g., importance or order). It is to be understood that if an element (e.g., a firstelement) is referred to, with or without the term "operatively" or "communicatively,"as "coupled with," "coupled to," "connected with," or "connected to" another element(e.g., a second element), it denotes that the element may be coupled with the otherelement directly (e.g., through wires), wirelessly, or via a third element.
[00176] As used in connection with various embodiments of the disclosure, theterm "module" may include a unit implemented in hardware, software, or firmware,and may interchangeably be used with other terms, for example, "logic," "logic block,""part," or "circuitry." A module may be a single integral component, or a minimumunit or part thereof, adapted to perform one or more functions. For example,according to an embodiment, the module may be implemented in a form of anapplication-specific integrated circuit (ASIC).
[00177] Various embodiments as set forth herein may be implemented assoftware (e.g., the program 140) including one or more instructions that are stored ina storage medium (e.g., internal memory 136 or external memory 138) that is readableby a machine (e.g., the electronic device 101). For example, a processor (e.g., theprocessor 120) of the machine (e.g., the electronic device 101) may invoke at least oneof the one or more instructions stored in the storage medium, and execute it, with orwithout using one or more other components under the control of the processor. Thisallows the machine to be operated to perform at least one function according to theat least one instruction invoked. The one or more instructions may include a codegenerated by a complier or a code executable by an interpreter. The machine-readablestorage medium may be provided in the form of a non-transitory storage medium.Wherein, the term "non-transitory" simply denotes that the storage medium is atangible device, and does not include a signal (e.g., an electromagnetic wave), but thisterm does not differentiate between where data is semi-permanently stored in thestorage medium and where the data is temporarily stored in the storage medium.
[00178] According to an embodiment, a method according to variousembodiments of the disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product between a sellerand a buyer. The computer program product may be distributed in the form of amachine-readable storage medium (e.g., compact disc read only memory (CD-ROM)),or be distributed (e.g., downloaded or uploaded) online via an application store (e.g.,PlayStoreTM), or between two user devices (e.g., smart phones) directly. If distributedonline, at least part of the computer program product may be temporarily generatedor at least temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store, or a relayserver.
[00179] According to various embodiments, each component (e.g., a module ora program) of the above-described components may include a single entity or multipleentities, and some of the multiple entities may be separately disposed in differentcomponents. According to various embodiments, one or more of the above-describedcomponents may be omitted, or one or more other components may be added.Alternatively or additionally, a plurality of components (e.g., modules or programs)may be integrated into a single component. In such a case, according to variousembodiments, the integrated component may still perform one or more functions ofeach of the plurality of components in the same or similar manner as they areperformed by a corresponding one of the plurality of components before theintegration. According to various embodiments, operations performed by the module,the program, or another component may be carried out sequentially, in parallel,repeatedly, or heuristically, or one or more of the operations may be executed in adifferent order or omitted, or one or more other operations may be added.
Claims
1. An electronic device comprising: a plurality of pixels; light-receiving pixels arranged between neighboring pixels among the plurality of pixels; a display composed of the plurality of pixels and the light-receiving pixels; and a processor operatively connected to the display, wherein the processor is configured to: receive light reflected from each of a user's left and right eyes through one or more light-receiving pixels constituting the display, based on the light received through the one or more light-receiving pixels, identify a gaze direction of each of the user's left and right eyes, and based on the gaze direction, output an image.
2. The electronic device of claim 1, wherein the processor is further configured to: receive light having been output from the display and reflected from each of the user's left and right eyes through the one or more light-receiving pixels, and adjust a timing of driving the plurality of pixels and a timing of driving the one or more light-receiving pixels, and wherein a path through which the light is output and a path through which the light is received through the one or more light-receiving pixels are the same.
3. The electronic device of claim 2, wherein the processor is further configured to: control the outputting of the image by driving the plurality of pixels on a first periodicity, and receive the light reflected from each of the user's left and right eyes by driving the one or more light-receiving pixels on a second periodicity equal to or longer than the first periodicity.
4. The electronic device of claim 2, wherein the processor is further configured to: control the outputting of the image by driving the plurality of pixels on a first periodicity, and receive the light reflected from each of the user's left and right eyes by driving the one or more light-receiving pixels on a second periodicity shorter than the first periodicity.
5. The electronic device of claim 1, further comprising: at least one light source configured to output infrared light, wherein the processor is further configured to: receive the infrared light output through the at least one light source and reflected from each of user's left and right eyes through the one or more light-receiving pixels, and wherein a path through which the infrared light is output and a path through which the infrared light is received through the one or more light-receiving pixels are different.
6. The electronic device of claim 1, wherein each of the plurality of pixels and the one or more light-receiving pixels includes a micro lens, and wherein the plurality of pixels and the one or more light-receiving pixels are separated by partition walls.
7. The electronic device of claim 1, wherein the display includes signal transmission paths for controlling the plurality of pixels and signal transmission paths for controlling the one or more light-receiving pixels.
8. The electronic device of claim 1, wherein the processor is further configured to: based on a charge amount of the one or more light-receiving pixels having received light reflected from each of the user's left and right eyes being less than or equal to a specified charge amount, receive the light reflected from each of the user's left and right eyes by using the one or more light-receiving pixels and another one or more light-receiving pixels arranged close to the one or more light-receiving pixels.
9. A method for obtaining user's gaze information by using light-receiving pixels of a display of an electronic device, the method comprising: receiving light reflected from each of a user's left and right eyes through one or more light-receiving pixels constituting the display; based on the light received through the one or more light-receiving pixels, identifying a gaze direction of each of the user's left and right eyes; and based on the gaze direction, outputting an image.
10. The method of claim 9, wherein the receiving of the light reflected from each of user's left and right eyes through the one or more light-receiving pixels comprises receiving light having been output from the display and reflected from each of the user's left and right eyes through the one or more light-receiving pixels, and wherein a path through which the light is output and a path through which the light is received through the one or more light-receiving pixels are the same.
11. The method of claim 10, further comprising: adjusting a timing for driving a plurality of pixels constituting the display and a timing for driving the one or more light-receiving pixels.
12. The method of claim 11, wherein the outputting of the image comprises outputting the image by driving the plurality of pixels on a first periodicity, and wherein the method further comprises: receiving the light reflected from each of the user's left and right eyes by driving the one or more light-receiving pixels on a second periodicity equal to or longer than the first periodicity.
13. The method of claim 11, wherein the outputting of the image comprises outputting the image by driving the plurality of pixels on a first periodicity, and wherein the method further comprises: receiving the light reflected from each of the user's left and right eyes by driving the one or more light-receiving pixels on a second periodicity shorter than the first periodicity.
14. The method of claim 9, wherein the receiving of the light reflected from each of user's left and right eyes through the one or more light-receiving pixels comprises receiving, through the one or more light-receiving pixels, infrared light having been output from at least one light source and reflected from each of user's left and right eyes, and wherein a path through which the infrared light is output and a path through which the infrared light is received through the one or more light-receiving pixels are different.
15. The method of claim 9, wherein the display includes signal transmission paths for controlling a plurality of pixels constituting the display and signal transmission paths for controlling the one or more light-receiving pixels.
