Wearable electronic device for tracking gaze and face

The integration of synchronized cameras and infrared lights in wearable electronic devices addresses the challenge of tracking user gaze and facial expressions in varying light conditions, enhancing the quality of augmented and virtual reality experiences.

WO2024237753A1PCT designated stage expired Publication Date: 2024-11-21SAMSUNG ELECTRONICS CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/095570
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-07-24
Filing Date
2024-03-15
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Current wearable electronic devices, such as VR and AR devices, face challenges in accurately tracking user gaze and facial expressions, especially in varying light conditions, which affects the quality of augmented and virtual reality experiences.

Method used

The implementation of a wearable electronic device equipped with multiple cameras and infrared lights that synchronize exposure times to prevent crosstalk and enhance image capture, allowing for precise gaze and facial expression tracking, even in low-light environments.

Benefits of technology

This solution enables more accurate and reliable tracking of user gaze and facial expressions, improving the overall quality of augmented and virtual reality experiences by ensuring consistent performance across different lighting conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024095570_21112024_PF_FP_ABST
    Figure KR2024095570_21112024_PF_FP_ABST
Patent Text Reader

Abstract

A wearable electronic device according to an embodiment may comprise: a processor; and a memory that stores instructions to be executed by the processor. The wearable electronic device may comprise: a first tracking device including first lights corresponding to a first area of a user wearing the wearable electronic device and first cameras corresponding to the first area; and a second tracking device including second lights corresponding to a second area of the user and second cameras corresponding to the second area. The memory may store one or more computer programs including computer-executable instructions that, when executed by the one or more processors, cause the wearable electronic device to execute an action of generating a first signal related to exposure of a first primary camera from among the first cameras before an exposure time of the first primary camera starts, and inputting, into the second cameras, the first signal as a signal notifying a start of a frame.
Need to check novelty before this filing date? Find Prior Art

Description

Wearable electronic devices that track gaze and face

[0001] Various embodiments relate to wearable electronic devices for tracking gaze and face.

[0002] A variety of wearable electronic devices, such as virtual reality (VR) devices, augmented reality (AR) devices, and / or mixed reality (MR) devices, are becoming commercially available. Wearable electronic devices, such as head-mounted displays (HMDs), are worn by users on their heads or eyes, and may take the form of glasses or goggles that include external lenses for viewing nearby objects or capturing images of objects encountered.

[0003] According to one embodiment, a wearable electronic device includes one or more processors, a memory storing instructions to be executed by the processors, a first tracking device including first lights corresponding to a first area of ​​a user wearing the wearable electronic device and first cameras corresponding to the first area, and a second tracking device including second lights corresponding to a second area of ​​the user and second cameras corresponding to the second area, wherein the memory may store one or more computer programs including computer-executable instructions that, when executed by the one or more processors, cause the wearable electronic device to perform an operation of inputting a first signal related to exposure of a first primary camera among the first cameras as a signal notifying the start of a frame to the second cameras.

[0004] According to one embodiment, a wearable electronic device includes one or more processors, a memory communicatively coupled to the one or more processors, a first tracking device including first infrared lights that generate images reflected to the left and right eyes of a user wearing the wearable electronic device, and a first primary camera that tracks the reflected images and the left eye and a first secondary camera that tracks the images reflected to the right eye of the user and the right eye of the user, and a second tracking device including second infrared lights that reflect lights to the face of the user, and second secondary cameras that recognize a facial expression of the user by the second infrared lights, wherein the memory may store one or more computer programs including computer-executable instructions that, when executed by the one or more processors, cause the wearable electronic device to perform an operation of generating a first signal related to exposure of the first primary camera before a first exposure time of the first primary camera begins and inputting the first signal as a signal notifying the start of a frame to the second secondary cameras.

[0005] FIG. 1 is a block diagram illustrating a configuration of an electronic device within a network environment according to one embodiment.

[0006] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0007] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device according to various embodiments.

[0008] FIGS. 4A and 4B are block diagrams of wearable electronic devices according to various embodiments.

[0009] FIG. 5 is a diagram showing the arrangement of cameras and lights of a first tracking device and a second tracking device in a wearable electronic device according to one embodiment.

[0010] FIG. 6 is a diagram illustrating a temporal relationship between a video frame and lighting used in a wearable electronic device according to one embodiment.

[0011] FIGS. 7A and 7B are diagrams for explaining input / output relationships between primary cameras and secondary cameras in a wearable electronic device according to various embodiments.

[0012] FIG. 8 is a diagram illustrating the operation between the optical drivers for the first lights of the first tracking device and the AP (Application Processor) in a wearable electronic device according to one embodiment.

[0013] FIG. 9 is a diagram illustrating an example of an arrangement between a first tracking device, a second tracking device, and optical drivers in a wearable electronic device according to one embodiment.

[0014] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0015] FIG. 1 is a block diagram illustrating a configuration of an electronic device within a network environment according to an embodiment. Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to an embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0016] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0017] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0018] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0019] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0020] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0021] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0022] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0023] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0024] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0025] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HMDI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0026] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HMDI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0027] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0028] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0029] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0030] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0031] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). For example, the wireless communication module (192) may use subscriber information stored in the subscriber identification module (e.g., an international mobile subscriber identity (IMSI)) to verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

[0032] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0033] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0034] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0035] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0036] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0037] According to one embodiment, each of the external electronic devices (102, 104) may be the same type of device as or a different type of device than the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of executing the function or service itself or in addition, request one or more external electronic devices to execute at least a part of the function or service. The one or more external electronic devices that receive the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a part of a response to the request.

[0038] For example, an external electronic device (102) can render content data executed in an application and transmit it to the electronic device (101), and the electronic device (101) that receives the data can output the content data to the display module (160). If the electronic device (101) detects a user's movement through a sensor, the processor (120) of the electronic device (101) can correct the rendering data received from the external electronic device (102) based on the movement information and output it to the display module (160). Alternatively, the processor (120) of the electronic device (101) can transmit the movement information to the external electronic device (102) and request rendering so that the screen data is updated accordingly. According to an embodiment, the external electronic device (102) may be various types of devices, such as a smartphone or a case device that can store and charge the electronic device (101).

[0039] Electronic devices according to embodiments disclosed herein may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments disclosed herein are not limited to the aforementioned devices.

[0040] Hereinafter, various embodiments will be described as an example in which the electronic device (101) is a wearable electronic device such as an HMD (head-mounted display or head-mounted device).

[0041] FIG. 2 is a diagram illustrating a structure in which an electronic device according to one embodiment is implemented in the form of a wearable electronic device.

[0042] Referring to FIG. 2, a wearable electronic device (200) according to one embodiment (e.g., electronic device (101) of FIG. 1) may be worn on a user's face and provide the user with images related to an augmented reality service and / or a virtual reality service.

[0043] In one embodiment, the wearable electronic device (200) includes a first display (205), a second display (210), a first screen display unit (215a), a second screen display unit (215b), a first input optical member (220a), a second input optical member (220b), a first transparent member (225a), a second transparent member (225b), a lighting unit (230a, 230b), a first PCB (235a), a second PCB (235b), a first hinge (240a), a second hinge (240b), a first camera (245a, 245b, 245c, 245d), a plurality of microphones (e.g., a first microphone (250a), a second microphone (250b), a third microphone (250c)), a plurality of speakers (e.g., a first speaker (255a), a second speaker (255b)), It may include a battery (260), a second camera (275a, 275b), a third camera (265), a visor (270a, 270b), a right eye light source (241, 242, 243, 261, 262, 263), and a left eye light source (251, 252, 253, 271, 272, 273).

[0044] In one embodiment, the displays (e.g., the first display (205) and the second display (210)) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not shown, when the display is made of one of the liquid crystal display, the digital mirror display, or the silicon liquid crystal display, the wearable electronic device (200) may include a light source that irradiates light to a screen output area of ​​the display. In another embodiment, when the display is capable of generating light on its own, for example, when it is made of one of the organic light emitting diode or the micro LED, the wearable electronic device (200) may provide a good quality virtual image to the user even without including a separate light source. In one embodiment, if the display is implemented with an organic light-emitting diode or micro LED, a light source is unnecessary, so the wearable electronic device (200) can be made lightweight. Hereinafter, a display that can generate light on its own is referred to as a self-luminous display, and the description will be made on the premise of a self-luminous display.

[0045] The displays according to various embodiments of the present document (e.g., the first display (205) and the second display (210)) may be composed of at least one micro LED (micro light emitting diode). For example, the micro LED can express red (R, red), green (G, green), and blue (B, blue) by self-luminescence, and is small in size (e.g., 100 μm or less), so that one chip can implement one pixel (e.g., one of R, G, and B). Accordingly, when the display is composed of the micro LED, it can provide a high resolution without a backlight unit (BLU).

[0046] Not limited thereto, one pixel may include R, G, and B, and one chip may be implemented with multiple pixels including R, G, and B.

[0047] In one embodiment, the display (e.g., the first display (205) and the second display (210)) may be composed of a display area composed of pixels for displaying a virtual image and light-receiving pixels (e.g., photo sensor pixels) arranged between the pixels for receiving light reflected from the eye, converting it into electrical energy, and outputting it.

[0048] In one embodiment, the wearable electronic device (200) can detect a user's gaze direction (e.g., eye movement) through light-receiving pixels. For example, the wearable electronic device (200) can detect and track a gaze direction for the user's right eye and a gaze direction for the user's left eye through one or more light-receiving pixels constituting the first display (205) and one or more light-receiving pixels constituting the second display (210). The wearable electronic device (200) can determine the position of the center of the virtual image based on the gaze directions of the user's right and left eyes (e.g., the direction in which the pupils of the user's right and left eyes are gazing) detected through one or more light-receiving pixels.

[0049] In one embodiment, the right eye light source (241, 242, 243, 261, 262, 263) and the left eye light source (251, 252, 253, 271, 272, 273) attached around the frame of the wearable electronic device (200) can be used as an auxiliary means to facilitate detection of the eye gaze when capturing the pupil with the second camera (275a, 275b). When the right eye light source (241, 242, 243, 261, 262, 263) and the left eye light source (251, 252, 253, 271, 272, 273) are used as auxiliary means for detecting the direction of gaze, they may be composed of a light emitting diode (LED) or an infrared ray light emitting diode (IR LED) that generates infrared wavelengths.

[0050] In one embodiment, light emitted from displays (e.g., the first display (205) and the second display (210)) may pass through a lens (not shown) and a waveguide to reach a first screen display unit (215a) formed on a first transparent member (225a) positioned to face the user's right eye and a second screen display unit (215b) formed on a second transparent member (225b) positioned to face the user's left eye. For example, light emitted from displays (e.g., the first display (205) and the second display (210)) may pass through a waveguide to be reflected by an input optical member (e.g., the first input optical member (220a) and the second input optical member (220b)) and a grating area formed on the screen display units (215a, 215b) and be transmitted to the user's eyes. The first transparent member (225a) and / or the second transparent member (225b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent.

[0051] In one embodiment, a lens (not shown) may be positioned in front of a display (e.g., a first display (205) and a second display (210)). The lens (not shown) may include a concave lens and / or a convex lens. For example, the lens (not shown) may include a projection lens or a collimation lens.

[0052] In one embodiment, the screen display unit (215a, 215b) or the transparent member (e.g., the first transparent member (225a), the second transparent member (225b)) may include a lens including a waveguide, a reflective lens.

[0053] In one embodiment, the waveguide may be made of glass, plastic, or polymer, and may include nano-patterns formed on one surface of the inner or outer surface, for example, a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one end of the waveguide may be propagated within the display waveguide by the nano-patterns and provided to the user. In one embodiment, a waveguide composed of a free-form prism may provide the incident light to the user through a reflective mirror. The waveguide may include at least one diffractive element, for example, a diffractive optical element (DOE), a holographic optical element (HOE), or a reflective element, for example, a reflective mirror. In one embodiment, the waveguide may guide light emitted from the display (205, 210) to the user's eyes by using at least one diffractive element or reflective element included in the waveguide.

[0054] According to various embodiments, the diffractive element may include an input optical member (220a, 220b) / output optical member (not shown). For example, the input optical member (220a, 220b) may mean an input grating area, and the output optical member (not shown) may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from a display (e.g., a first display (205) and a second display (210)) (e.g., a micro LED) to transmit the light to a transparent member (e.g., a first transparent member (250a), a second transparent member (250b)) of a screen display unit (215a, 215b). The output grating region can serve as an outlet for diffracting (or reflecting) light transmitted to the transparent member of the waveguide (e.g., the first transparent member (250a), the second transparent member (250b)) toward the user's eyes.

[0055] According to various embodiments, the reflective element may include a total internal reflection (TIR) ​​optical element or a total internal reflection waveguide. For example, total internal reflection may refer to a method of guiding light such that light (e.g., a virtual image) entering through an input grating region is 100% reflected from one surface (e.g., a specific surface) of the waveguide, thereby transmitting 100% of the light to the output grating region.

[0056] In one embodiment, light emitted from a display (205, 210) may be guided along an optical path to a waveguide via input optical elements (220a, 220b). Light traveling within the waveguide may be guided toward a user's eyes via output optical elements. The screen display (215a, 215b) may be determined based on the light emitted toward the user's eyes.

[0057] In one embodiment, the first camera (245a, 245b, 245c, 245d) may be used to detect and track the face of the other party, the head of the other party, and / or the hand of the user, and / or to recognize the gesture (e.g., hand motion) of the user. The first camera (245a, 245b, 245c, 245d) may be used for 3 degrees of freedom (3DoF), 6DoF face or head tracking, hand detection and tracking, gesture, position (spatial, environmental) recognition, and / or movement recognition. For example, the first camera (245a, 245b, 245c, 245d) may include a GS (global shutter) camera to detect and track the movement of the head and hand including the face. Depending on the embodiment, a third camera (265) may be used for hand detection and tracking and user gesture recognition.

[0058] For example, the first camera (245a, 245b, 245c, 245d) may be a stereo camera for head tracking and spatial recognition including a face, and a camera of the same standard and performance may be applied.

[0059] The first camera (245a, 245b, 245c, 245d) may be a GS camera with excellent performance (e.g., image drag) to detect and track fine movements such as quick hand movements and fingers.

[0060] According to various embodiments, the first camera (245a, 245b, 245c, 245d) may be a rolling shutter (RS) camera. The first camera (245a, 245b, 245c, 245d) may perform spatial recognition for 6 DoF and SLAM (simultaneous localization and mapping) functions through depth shooting. The first camera (245a, 245b, 245c, 245d) may perform a user gesture recognition function.

[0061] In one embodiment, the second camera (275a, 275b) can capture, recognize (detect), and / or track the trajectory of the user's eye (e.g., pupil, iris) or gaze. The second camera (275a, 275b) can periodically or aperiodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1). The second camera (275a, 275b) can also capture external images. The second camera (275a, 275b) may be referred to as an 'ET (eye tracking) camera.' The second camera (275a, 275b) can track the direction of the user's gaze. The wearable electronic device (200) can position the center of a virtual image projected on the screen display unit (215a, 215b) according to the direction in which the user's eyes are looking, based on the user's gaze direction.

[0062] The second camera (275a, 275b) for tracking the gaze direction may be a GS camera that can detect the user's pupil and track rapid eye movements. The second cameras (275a, 275b) may be installed for the left and right eyes, respectively, and the second camera (275a) for the right eye and the second camera (275b) for the left eye may be cameras with the same performance and specifications.

[0063] In one embodiment, the third camera (265) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. The third camera (265) may include a color camera equipped with functions for obtaining high-quality images, such as an AF (auto focus) function and an OIS (optical image stabilizer). However, the third camera (265) is not limited thereto, and may include a GS (global shutter) camera or an RS (rolling shutter) camera.

[0064] In one embodiment, the fourth camera (280a, 280b, 280c) can recognize the user's face and / or facial expression. The fourth camera (280a, 280b, 280c) may be referred to as a 'face tracking (FT) camera.'

[0065] In one embodiment, at least one sensor (e.g., a gyroscope sensor, an acceleration sensor, a magnetometer sensor, a touch sensor, an ambient light sensor, and / or a gesture sensor), and a first camera (245a, 245b, 245c, 245d) can perform at least one of head tracking for 6DoF, pose estimation & prediction, gesture and / or spatial recognition, and SLAM with depth shooting.

[0066] In one embodiment, the first cameras (245a, 245b, 245c, 245d) may be used separately as cameras for head tracking including the face and cameras for hand tracking. Not all of the first cameras (245a, 245b, 245c, 245d) must be used, and only some of them (e.g., 245a and 245b or 245c, 245d) may be used.

[0067] According to one embodiment, at least one of the first camera (245a, 245b, 245c, 245d), the second camera (275a, 275b), and the third camera (265) may be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0068] In one embodiment, the lighting units (230a, 230b) may have different uses depending on the attachment location. For example, the lighting units (230a, 230b) may be attached together with the first camera (245a, 245b, 245c, 245d) mounted around a hinge (e.g., a first hinge (240a), a second hinge (240b)) connecting a frame and a temple or around a bridge connecting the frames. When shooting with a GS camera, the lighting units (230a, 230b) may be used as a means of supplementing the surrounding brightness. For example, the lighting units (230a, 230b) may be used when it is difficult to detect a subject to be shot due to a dark environment or mixing or reflecting light from multiple light sources.

[0069] In one embodiment, a PCB (e.g., a first PCB (235a), a second PCB (235b)) may include a processor (e.g., a processor (120) of FIG. 1), a memory (e.g., a memory (130) of FIG. 1), and a communication module (e.g., a communication module (190) of FIG. 1) that control components of a wearable electronic device (200).

[0070] For example, the communication module may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the wearable electronic device (200) and an external electronic device, and the performance of communication through the established communication channel. The PCB (e.g., the first PCB (235a), the second PCB (235b)) may transmit electrical signals to components constituting the wearable electronic device (200).

[0071] A communication module (e.g., the communication module (190) of FIG. 1) may include one or more communication processors that operate independently from the processor and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device via a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0072] The wireless communication module can support 5G networks and next-generation communication technologies beyond 4G networks, such as new radio access technology (NR). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module can support high-frequency bands (e.g., mmWave bands) to achieve high data rates, for example. The wireless communication module may support various technologies to secure performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna.

[0073] The wearable electronic device (200) may further include an antenna module (e.g., the antenna module (197) of FIG. 1). The antenna module may transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a first PCB (235a), a second PCB (235b)). According to one embodiment, the antenna module may include a plurality of antennas (e.g., an array antenna).

[0074] In one embodiment, multiple microphones (e.g., a first microphone (250a), a second microphone (250b), and a third microphone (250c)) can process external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function being performed (or the application being executed) on the wearable electronic device (200).

[0075] In one embodiment, a plurality of speakers (e.g., a first speaker (255a), a second speaker (255b)) (e.g., an audio output module (155) of FIG. 1) can output audio data received from a communication module or stored in a memory.

[0076] In one embodiment, one or more batteries (260) (e.g., battery (189) of FIG. 1) may be included and may supply power to components that make up the wearable electronic device (200).

[0077] In one embodiment, the visor (270a, 270b) can adjust the amount of external light transmitted into the user's eyes according to the transmittance. The visor (270a, 270b) can be positioned in front or behind the screen display unit (215a, 215b). The front of the screen display unit (215a, 215b) can mean the direction opposite to the user wearing the wearable electronic device (200), and the back can mean the direction toward the user wearing the wearable electronic device (200). The visor (270a, 270b) can protect the screen display unit (215a, 215b) and adjust the amount of external light transmitted.

[0078] FIGS. 3A and 3B are diagrams illustrating the front and back of a wearable electronic device (300) according to various embodiments. FIG. 3A may be an external appearance of the wearable electronic device (300) when viewed in a first direction (①), and FIG. 3B may be an external appearance of the wearable electronic device (300) when viewed in a second direction (②). When a user wears the wearable electronic device (300), the external appearance viewed by the user's eyes may be FIG. 3B.

[0079] Referring to FIG. 3A, according to various embodiments, the electronic device (101) of FIG. 1 may include a wearable electronic device (300) that provides a service that provides an extended reality (XR) experience to a user. For example, XR or XR service may be defined as a service that collectively refers to virtual reality (VR), augmented reality (AR), and / or mixed reality (MR).

[0080] According to one embodiment, the wearable electronic device (300) may refer to a head-mounted device or a head-mounted display worn on the user's head, but may also be configured in the form of at least one of glasses, goggles, a helmet, or a hat. The wearable electronic device (300) may include an OST (optical see-through) type configured to allow external light to reach the user's eyes through the glasses when worn, or a VST (video see-through) type configured to allow light emitted from the display to reach the user's eyes when worn, but block external light so that external light does not reach the user's eyes.

[0081] According to one embodiment, the wearable electronic device (300) may be worn on the user's head and provide the user with an image related to an extended reality (XR) service. For example, the wearable electronic device (300) may provide XR content (hereinafter referred to as an XR content image) that outputs at least one virtual object to be superimposed on a display area or an area determined to be the user's field of view (FoV). According to one embodiment, the XR content may refer to an image or image related to a real space acquired through a camera (e.g., a camera for taking pictures) or an image or image in which at least one virtual object is superimposed on a virtual space. According to one embodiment, the wearable electronic device (300) may provide XR content based on a function being performed by the wearable electronic device (300) and / or a function being performed by one or more external electronic devices (e.g., the external electronic devices 102, 104, or 108 of FIG. 1).

[0082] According to one embodiment, the wearable electronic device (300) is at least partially controlled by an external electronic device (e.g., the external electronic devices (102 or 104) of FIG. 1), and may perform at least one function under the control of the external electronic device, but may also perform at least one function independently.

[0083] Referring to FIG. 3A, cameras (e.g., cameras for second functions (311, 312), cameras for first functions (315)) and / or depth sensors (317)) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be placed on the first surface (310) of the main body housing of the wearable electronic device (300).

[0084] In one embodiment, the second function cameras (311, 312) can obtain images related to the surrounding environment of the wearable electronic device (300). The first function cameras (315) can obtain images when the wearable electronic device is worn by the user. The first function cameras (315) can be used for hand detection and tracking, and recognition of user gestures (e.g., hand movements). The first function cameras (315) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the second function cameras (311, 312) can also be used for hand detection and tracking, and user gestures.

[0085] In one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (317), the first camera (245a, 245b, 245c, 245d) may determine the distance to an object.

[0086] Referring to FIG. 3b, a fourth functional camera (e.g., a face recognition camera) (325, 326, 327) (e.g., the fourth camera (280a, 280b, 280c) of FIG. 2) and / or a display (321) (and / or a lens) may be arranged on the second surface (320) of the main body housing.

[0087] In one embodiment, a face recognition camera (325, 326, 327) adjacent to the display may be used to recognize the user's face, or may recognize and / or track the user's two eyes.

[0088] In one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include some of the plurality of cameras (315). Although not shown in FIGS. 3A and 3B , the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2 .

[0089] According to one embodiment, a wearable electronic device (300) includes a main body that mounts at least some of the components of FIG. 1, and a display (321) (e.g., display module (160) of FIG. 1), a third function camera (e.g., a camera for eye tracking) (328a, 328b), and a fourth function camera (e.g., a camera for face recognition) (325, 326, 327) may be arranged in a first direction (①) facing the user's face in the main body.

[0090] A first function camera (e.g., a recognition camera) (315), a second function camera (e.g., a shooting camera) (311, 312), a depth sensor (317), and a touch sensor (313) may be arranged in a second direction (②) opposite to the first direction (①) of the main body. Although not shown in the drawing, the main body may include a memory (e.g., a memory (130) of FIG. 1) and a processor (e.g., a processor (120) of FIG. 1), and may further include other components shown in FIG. 1.

[0091] According to one embodiment, the display (321) may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0092] In one embodiment, when the display (321) is formed of one of a liquid crystal display (LCD), a digital mirror display, or a silicon liquid crystal display (LCD), the wearable electronic device (300) may include a light source that irradiates light to a screen output area of ​​the display (321). In another embodiment, when the display (321) can generate light on its own, for example, when the wearable electronic device (300) is formed of one of an organic light emitting diode (OLED) or a micro LED, the wearable electronic device (300) may provide a good quality XR content image to the user even without including a separate light source. In one embodiment, when the display (321) is formed of an organic light emitting diode (OLED) or a micro LED, a light source is unnecessary, and thus the electronic wearable electronic device (300) may be lightweight.

[0093] According to one embodiment, the display (321) may include a first display (321a) and / or a second display (321b). The user may use the wearable electronic device (300) while wearing it on his or her face. The first display (321a) and / or the second display (321b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. The first display (321a) may include a first transparent member, and the second display (321b) may include a second transparent member. According to one embodiment, the first display (321a) may be arranged to face the user's right eye in the third direction (③), and the second display (321b) may be arranged to face the user's left eye in the fourth direction (④). According to various embodiments, if the display (321) is transparent, it may be arranged at a position facing the user's eyes to configure a screen display area.

[0094] According to one embodiment, the display (321) may include a lens including a transparent waveguide. The lens may serve to adjust the focus so that the screen (e.g., XR content image) output to the display (321) can be viewed by the user's eyes. For example, light emitted from the display panel may pass through the lens and be transmitted to the user through a waveguide formed within the lens. The lens may be configured as a Fresnel lens, a pancake lens, or a multi-channel lens.

[0095] An optical waveguide (e.g., a waveguide) can serve to transmit light generated by a display (e.g., a display (205, 210) of FIG. 2) to a user's eye. The optical waveguide can be made of glass, plastic, or a polymer, and can include nano-patterns formed on a portion of an inner or outer surface, for example, a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one end of the optical waveguide, that is, an output image of the display (e.g., a display (205, 210) of FIG. 2), can be propagated within the optical waveguide and provided to the user. In addition, an optical waveguide composed of a free-form prism can provide the incident light to the user through a reflective mirror. The optical waveguide can include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). An optical waveguide can guide an image output from a display (e.g., display (205, 210) of FIG. 2) to a user's eye by using at least one diffractive element or reflective element included in the optical waveguide.

[0096] According to one embodiment, the diffractive element may include an input optical member / output optical member (not shown). For example, the input optical member may mean an input grating area, and the output optical member (not shown) may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from a light source (e.g., Micro LED) to transmit the light to a transparent member (e.g., first display (321a), second display (321b)) of a screen display area. The output grating area may serve as an outlet that diffracts (or reflects) light transmitted to a transparent member (e.g., first transparent member, second transparent member) of an optical waveguide to a user's eye.

[0097] In some embodiments, the reflective element may comprise a total internal reflection (TIR) ​​optical element or waveguide for total internal reflection. For example, total internal reflection may refer to a method of directing light such that light (e.g., a virtual image) entering through an input grating region is reflected substantially 100% from one surface (e.g., a specific surface) of the optical waveguide, thereby transmitting substantially 100% of the light to the output grating region at an angle of incidence.

[0098] In one embodiment, light emitted from the display (321) may be guided along an optical path through an input optical element into a waveguide. Light traveling within the optical waveguide may be guided toward the user's eyes through an output optical element. The screen display area may be determined based on the light emitted toward the user's eyes.

[0099] The first function camera (e.g., recognition camera) (315) can be used for the purpose of detecting user movement or recognizing user gestures. The first function camera (315) can support at least one of head tracking, hand detection and hand tracking, and spatial recognition. For example, the first function camera (315) mainly uses a GS (global shutter) camera, which has superior performance compared to an RS (rolling shutter) camera, to detect and track fine movements of hand movements and fingers, and can be configured as a stereo camera including two or more GS cameras for head tracking and spatial recognition. The first function camera (315) can perform a SLAM (simultaneous localization and mapping) function to recognize information (e.g., location and / or direction) related to the surrounding space through spatial recognition for 6DoF and depth shooting.

[0100] A second function camera (e.g., a shooting camera) (311, 312) can be used to capture the outside and generate an image or video corresponding to the outside and transmit it to a processor (e.g., a processor (120) of FIG. 1). The processor (120) can display the image provided from the second function camera (311, 312) on a display (321). The second function camera (311, 312) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. For example, the second function camera (311, 312) may include a color camera equipped with functions for obtaining high-quality images, such as an AF (auto focus) function and an optical image stabilizer (OIS), but is not limited thereto, and the second function camera (311, 312) may also include a GS camera or an RS camera.

[0101] The third function camera (e.g., gaze tracking camera) (328a, 328b) may be positioned on the display (321) (or inside the main body) so that the camera lens faces the user's eyes when the user wears the wearable electronic device (300). The third function camera (328a, 328b) may be used for the purpose of detecting and tracking (ET, eye tracking) pupils. The processor (120) may track the movements of the user's left and right eyes in the images received from the third function camera (328a, 328b) to determine the gaze direction. The processor (120) may track the position of the pupil in the images so that the center of the XR content image displayed in the screen display area may be positioned according to the direction in which the pupil is looking. As an example, the third function camera (328a, 328b) may be a GS camera to detect the pupil and track the movement of the pupil. The third function camera (328a, 328b) can be installed for the left eye and the right eye, respectively, and cameras with the same performance and specifications can be used.

[0102] The fourth function camera (e.g., face recognition camera) (325, 326, 327) may be used to detect and track (FT) the user's facial expression when the user wears the wearable electronic device (300). According to one embodiment, the wearable electronic device (300) may include a lighting unit (e.g., LED) (not shown) as an auxiliary means for the cameras. As an example, the third function camera (328a, 328b) may use lighting included in the display so that the emitted light (e.g., IR LED of infrared wavelength) is directed toward the user's both eyes as an auxiliary means to facilitate gaze detection when tracking eye movement. As another example, the second function camera (311, 312) may further include a lighting unit (e.g., flash) as an auxiliary means to supplement the surrounding brightness when taking external pictures.

[0103] According to one embodiment, a depth sensor (or depth camera) (317) may be used to determine the distance to an object (e.g., an object), such as time of flight (TOF). Time of flight (TOF) is a technology that measures the distance to an object using a signal (e.g., near-infrared, ultrasound, or laser). After a signal is transmitted from a transmitter, a signal is measured at a receiver, and the distance to an object can be measured based on the flight time of the signal.

[0104] According to one embodiment, the touch sensor (313) may be arranged in the second direction (②) of the main body. For example, when a user wears the wearable electronic device (300), the user's eyes may look in the first direction (①) of the main body. The touch sensor (313) may be implemented as a single type or a type separated into left and right sides depending on the shape of the main body, but is not limited thereto. For example, when the touch sensor (313) is implemented as a type separated into left and right sides as illustrated in FIG. 3A, when a user wears the wearable electronic device (300), the first touch sensor (313a) may be arranged at the user's left eye position, such as in the fourth direction (④), and the second touch sensor (313b) may be arranged at the user's right eye position, such as in the third direction (③).

[0105] The touch sensor (313) can recognize a touch input using at least one of, for example, a capacitive, pressure-sensitive, infrared, or ultrasonic method. For example, the capacitive touch sensor (313) can recognize a physical touch (or contact) input or a hovering input (or proximity) of an external object. According to some embodiments, the wearable electronic device (300) may utilize a proximity sensor (not shown) to enable proximity recognition of an external object.

[0106] According to one embodiment, the touch sensor (313) has a two-dimensional surface and can transmit touch data (e.g., touch coordinates) of an external object (e.g., a user's finger) that comes into contact with the touch sensor (313) to the processor (120). The touch sensor (313) can detect a hovering input for an external object (e.g., a user's finger) that approaches within a first distance from the touch sensor (313), or detect a touch input that touches the touch sensor (313).

[0107] According to one embodiment, the touch sensor (313) may provide two-dimensional information about the point of contact as “touch data” to the processor (120) when an external object touches the touch sensor (313). The touch data may be described as a “touch mode.” The touch sensor (313) may provide hovering data about the time or location of hovering around the touch sensor (313) to the processor (120) when an external object is located within a first distance from the touch sensor (or in proximity, hovering above the touch sensor). The hovering data may be described as a “hovering mode / proximity mode.”

[0108] According to one embodiment, the wearable electronic device (300) may obtain hovering data using at least one of a touch sensor (313), a proximity sensor (not shown), or / and a depth sensor (317) to generate information about a distance, location, or time point between the touch sensor (313) and an external object.

[0109] According to one embodiment, the interior of the main body may include components of FIG. 1, for example, a processor (120) and a memory (130).

[0110] The memory (130) can store various instructions that can be performed by the processor (120). The instructions can include arithmetic and logical operations, data movement, or control commands such as input / output that can be recognized by the processor (120). The memory (130) can temporarily or permanently store various data, including volatile memory (e.g., volatile memory (132) of FIG. 1) and non-volatile memory (e.g., non-volatile memory (134) of FIG. 1).

[0111] The processor (120) may be operatively, functionally, and / or electrically connected to each component of the wearable electronic device (300), and may be configured to perform operations or data processing related to control and / or communication of each component. Operations performed by the processor (120) may be stored in the memory (130) and, when executed, may be executed by instructions that cause the processor (120) to operate.

[0112] Hereinafter, the computational and data processing functions that the processor (120) can implement on the wearable electronic device (300) are not limited, but a series of operations related to the XR content service function will be described. The operations of the processor (120) described below can be performed by executing instructions stored in the memory (130).

[0113] According to one embodiment, the processor (120) may generate a virtual object based on virtual information based on image information. The processor (120) may output a virtual object related to an XR service together with background space information through the display (321). For example, the processor (120) may capture an image related to an actual space corresponding to the field of view of a user wearing the wearable electronic device (300) through the second function camera (311, 312) to obtain image information or generate a virtual space for a virtual environment. For example, the processor (120) may control the display (321) to display XR content (hereinafter referred to as an XR content screen) in which at least one virtual object is output to be overlapped in an area determined to be a display area or a field of view (FoV) of the user.

[0114] According to one embodiment, a wearable electronic device (300) may have a form factor for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured to a body part of the user. The wearable electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0115] FIG. 4A is a block diagram of a wearable electronic device according to an embodiment. Referring to FIG. 4A, a wearable electronic device (400) according to an embodiment (e.g., the wearable electronic device (200) of FIG. 2 and / or the wearable electronic device (300) of FIG. 3) may include one or more processors (410) (e.g., the processor (120) of FIG. 1), memories (430) (e.g., the memory (130) of FIG. 1), a first tracking device (450), and a second tracking device (470). The wearable electronic device (400) may correspond to an electronic device that can be worn by a user, such as a head mounted display (HMD). A head mounted display may allow a user to feel a virtual image as if it were real through vivid images, videos, voices, etc.

[0116] A wearable electronic device (400) is worn on a user's face and can provide the user with images related to augmented reality services, virtual reality services, and / or mixed reality services. In one embodiment, the wearable electronic device (400) may include cameras (453, 473) to provide the user with augmented reality services, virtual reality services, and / or mixed reality services. The cameras (453, 473) detect wavelengths in the visible light range and the infrared range to acquire image frames, and the wearable electronic device (400) can perform gaze tracking and face tracking and recognize space using the image frames acquired by the cameras (453, 473).

[0117] One or more processors (410), memory (430), first tracking device (450), and second tracking device (470) may be connected to each other via a communication bus (405).

[0118] One or more processors (410) can control a first primary camera (e.g., the first primary camera (461) of FIG. 4b) among the first cameras (453) (e.g., the second cameras (275a, 275b) of FIG. 2 and / or the third functional cameras (328a, 328b) of FIG. 3)) to generate a first signal related to the exposure of the first primary camera and input the generated first signal as a signal notifying the start of a frame to the second cameras (473) (e.g., the fourth cameras (280a, 280b, 280c) of FIG. 2 and / or the fourth functional cameras (325, 326, 327) of FIG. 3). The first primary camera can be any one of the first cameras (453).

[0119] Accordingly, the first primary camera may generate a first signal (e.g., the first signal (621) of FIG. 6) related to the exposure of the first primary camera among the first cameras (453), and input the first signal as a signal notifying the start of a frame to the second cameras (473). Here, the 'first signal related to the exposure of the first primary camera' may correspond to, for example, a signal notifying the start of the first exposure time of the first primary camera, or a signal capable of notifying the exposure status of the first primary camera. At this time, the signal notifying the start of the first exposure time may be activated in accordance with the first exposure time of the first primary camera, or may be activated before the first exposure time. The 'first signal related to the exposure of the first primary camera' may have the form of a 'strobe signal'.

[0120] The first primary camera can input a first signal at the first exposure time of the first primary camera as a signal to notify the second cameras (473) of the start of a frame. For example, the first primary camera can input a first signal before the start of the first exposure time of the first primary camera as a signal to notify the second cameras (473) of the start of a frame. The signal to notify the second cameras (473) of the start of a frame can be, for example, an 'Fsync' signal.

[0121] The first primary camera can adjust the timing of generation of the first signal so that the first exposure time of the first primary camera and the second exposure times of the second cameras (473) do not overlap with each other. The one or more processors (410) can adjust the timing of generation of the first signal so that, for example, the interval between the first exposure time and the second exposure time is reduced in proportion to the transmission time of one data frame transmitted by the first primary camera. The first primary camera can adjust the timing of generation of the first signal so that, for example, the interval between the first exposure time and the second exposure time is less than half the transmission time of one data frame transmitted by the first primary camera. The one or more processors (410) can control the first primary camera to adjust the timing of synchronization with the remaining cameras (e.g., the first secondary camera and / or the second cameras (473)). The method by which the one or more processors (410) adjust the timing of generation of the first signal is described in more detail with reference to FIG. 6 below.

[0122] One or more processors (410) may input a third signal (e.g., the third signal (635) of FIG. 6) related to the exposure of the second cameras (473) as a second trigger signal corresponding to the second light driver for the second lights (471). One or more processors (410) may input, for example, third signals related to the exposure of each of the second-1 secondary camera and the second-2 secondary camera among the second cameras (473), as a second trigger signal corresponding to the second light driver for the second lights (471). The term 'primary / secondary camera' described herein may also be expressed as 'master / slave camera'.

[0123] The memory (430) may store instructions to be executed by one or more processors (410). The instructions may be configured to cause one or more processors (410) to execute various operations described above.

[0124] The first tracking device (450) may include first lights (451) and first cameras (453). The first lights (451) may correspond to a first area of ​​a user wearing the wearable electronic device (400). The first area may correspond to the user's eyes (both eyes), that is, the user's left eye and the user's right eye, but is not necessarily limited thereto. The first lights (451) may generate images reflected in the first area (e.g., the user's left eye and the user's right eye wearing the wearable electronic device (400).

[0125] In one embodiment, the wearable electronic device (400) may control the transmittance of the display to be lower as the brightness of the acquired image frame becomes brighter, so as to prevent the user's eyes from being dazzled by ambient light.

[0126] Since the wearable electronic device (400) acquires image frames using the cameras (453, 473), if the brightness around the cameras (453, 473) is dark, it may be difficult to perform gaze tracking, face tracking, and spatial recognition using the acquired image frames. In one embodiment, the wearable electronic device (400) may include infrared lights (e.g., first lights (451) and second lights (471)) to acquire image frames with the brightness required for gaze tracking, face tracking, and spatial recognition. If the brightness around the wearable electronic device (400) is dark, the wearable electronic device (400) may turn on the infrared lights to secure more light.

[0127] The first lights (451) may be, for example, light emitting diodes (LEDs) or infrared ray light emitting diodes (IR LEDs), but are not necessarily limited thereto.

[0128] The first lights (451) can turn on the light source for, for example, 2 msec, according to a control signal of the first light drivers (e.g., the 1-1 light driver for the left-eye lights and the 1-2 light driver for the right-eye lights), but is not necessarily limited thereto.

[0129] In one embodiment, the cameras (453, 473) detect wavelengths in the visible light range and the infrared range to acquire image frames. Therefore, when acquiring image frames while the infrared lights are turned on, the brightness of the acquired image frames may be bright even if the surrounding environment perceived by an actual person is dark. In order for the wearable electronic device (400) to check the surrounding brightness perceived by the user while the infrared lights are turned on, a separate light sensor may be required.

[0130] A wearable electronic device (400) according to one embodiment can detect ambient brightness without being affected by infrared lighting even without a light sensor by turning off infrared lighting in an image frame that is not used for eye tracking and face tracking among multiple image frames acquired through cameras (453, 473) and acquiring an image frame, and checking the ambient brightness using the acquired image frame.

[0131] The first cameras (453) may correspond to the first area. The first cameras (453) may include, for example, a first primary camera (461) and a first secondary camera (462) as illustrated in FIG. 4B. The first primary camera (461) may track images reflected on the user's left eye and the user's left eye. Here, "tracking the user's left eye" may be understood comprehensively to mean tracking not only the user's left eye but also the gaze of the left eye. When the first primary camera (461) receives an activation signal from one or more processors (410), it may transmit a trigger signal to the first-first light driver for the left-eye lights among the first lights (451).

[0132] The first secondary camera can track the images reflected from the user's right eye and the user's right eye. Similarly, "tracking the user's right eye" can be understood broadly to mean tracking not only the user's right eye but also the gaze of the user's right eye.

[0133] The first cameras (453), for example, the first primary camera (461) and the first secondary camera (462), may be synchronized with each other. Here, the first cameras (e.g., the first primary camera (461) and the first secondary camera (462)) being "synchronized" with each other means that the first primary camera (461) and the first secondary camera (462) start shooting at the same time, which may mean that the exposure times of the first primary camera (461) and the first secondary camera (462) are the same. In addition, the first primary camera and the first secondary camera being "synchronized" with each other may also be understood to mean that the first lights (451) for the first primary camera (461) and the first lights (451) for the first secondary camera (462) are "turned on" or "on" at the same time.

[0134] The second tracking device (470) may include second lights (471) and second cameras (473). The second lights (471) may correspond to a second area of ​​the user. The second area may correspond to the user's face, but is not necessarily limited thereto. The second lights (471) may reflect light onto the second area (e.g., the user's face). The second lights (471) may be, but are not necessarily limited to, light-emitting diodes (LEDs) or infrared light-emitting diodes (IR LEDs), for example.

[0135] The second cameras (473) may correspond to the second area. The second cameras (473) may recognize the user's facial expressions using the second lights. The second cameras (473) may be synchronized with each other.

[0136] The second cameras (473) may also be referred to as “second secondary cameras (e.g., the second secondary cameras (481) of FIG. 4B)” in that they operate based on a first signal related to the exposure of the first primary camera (461). The second cameras (473) may include a 2-1 secondary camera, a 2-2 secondary camera, and a 2-3 secondary camera. For example, the 2-1 secondary camera may capture the left side of the user’s face. The 2-2 secondary camera may capture the center of the user’s face. The 2-3 secondary camera may capture the right side of the user’s face.

[0137] An example of the arrangement between the first lights (451) and the first cameras (453) and the second lights (471) and the second cameras (473) can be seen in FIG. 4b below.

[0138] According to one embodiment, in order to implement an avatar system that uses both eye tracking and face tracking, the first tracking device (450) and the second tracking device (470) may operate in the same cycle. The first tracking device (450) and the second tracking device (470) may operate with a constant delay (e.g., the constant delay time (640) of FIG. 6) within the same cycle. The meaning that the first tracking device (450) and the second tracking device (470) operate with a constant delay time will be described in more detail with reference to FIG. 6 below.

[0139] FIG. 4B is a block diagram of a wearable electronic device according to an embodiment. Referring to FIG. 4B, a wearable electronic device (400) according to an embodiment (e.g., the wearable electronic device (200) of FIG. 2 and / or the wearable electronic device (300) of FIGS. 3A and 3B) may include one or more processors (420), a memory (440), a first tracking device (460), and a second tracking device (480). The one or more processors (420), the memory (440), the first tracking device (460), and the second tracking device (480) may be connected to each other via a communication bus (e.g., the communication bus (405) of FIG. 4A).

[0140] One or more processors (420) may cause the first primary camera (461) to generate a first signal related to exposure of the first primary camera (461) before the first exposure time of the first primary camera (461) begins, and input the signal as an indication of the start of a frame to the second secondary cameras (481) (e.g., the fourth cameras (280a, 280b, 280c) of FIG. 2 and / or the fourth functional cameras (325, 326, 327) of FIGS. 3A and 3B). A processor (420) according to one embodiment may be a secondary processor (e.g., secondary processor (123) of FIG. 1), which is connected to various sensors such as a camera (e.g., camera module (180) of FIG. 1) and / or components such as a display (e.g., display module (160) of FIG. 1), and may be a processor (e.g., Real-time IO processor) for supporting real-time processing operations.

[0141] The first primary camera (461) may be triggered or activated to transmit a first signal, for example, when generating a user's facial expression, when generating an avatar, and / or when the user wears a wearable device. The first primary camera (461) may transmit the first signal, for example, every frame.

[0142] The first primary camera (461) may receive a signal requesting transmission of a first signal from one or more processors (420) (e.g., an application processor (AP)), or may receive a signal requesting cessation of transmission of the first signal.

[0143] The memory (440) may store instructions to be executed by one or more processors (420). The instructions may be configured to cause one or more processors (420) to execute various operations described above.

[0144] The first tracking device (460) may include a first primary camera (461), a first secondary camera (462), a first-first optical driver (463), a first-second optical driver (464), and first infrared lights (465, 466). The first primary camera (461) may track images reflected on the left eye of a user wearing the wearable electronic device (400) and / or the left eye (or the gaze of the left eye) of the user. The first secondary camera (462) may track the right eye (or the gaze of the right eye) of the user. The first-first optical driver (463) may control the left eye lights (L1, L2, L3, L4, L5) (465) among the first infrared lights (465, 466) according to a second signal related to the exposure of the first secondary camera (462). The first-second optical driver (464) can control the right-eye lights (R1, R2, R3, R4, R5) (466) among the first infrared lights (465, 466) according to a second signal related to the exposure of the first secondary camera (462). The first infrared lights (465, 466) can generate images reflected on the left and right eyes of a user wearing the wearable electronic device (400). In the embodiment of FIG. 4b, the number of the first infrared lights (465, 466) is described as 10 as an example, but is not necessarily limited thereto. The number of the first infrared lights (465, 466) can be configured in various ways as needed.

[0145] The second tracking device (480) may include second secondary cameras (481), a second optical driver (483), and second infrared lights (L (485a), M (485b), R (485c)) (485).

[0146] The second secondary cameras (481) can recognize the user's facial expressions using the second infrared lights (485). The second secondary cameras (481) can include a second-first secondary camera, a second-second secondary camera, and a second-third secondary camera. The second-first secondary camera can capture the left side of the user's face. The second-second secondary camera can capture the center of the user's face. The second-third secondary camera can capture the right side of the user's face.

[0147] The second optical driver (483) can control the second infrared lights (485) according to a third signal related to the exposure of the second secondary cameras (481).

[0148] The second infrared lights (L(485a), M(485b), R(485c))(485) can reflect light onto the user's face. The second infrared light (L)(485a) can reflect light onto the left side of the user's face. The second infrared light (M)(485b) can reflect light onto the center of the user's face (e.g., between the eyebrows). The second infrared light (R)(485c) can reflect light onto the right side of the user's face.

[0149] The memories (430, 440) of FIGS. 4A and 4B can store at least one program including the application described below. In addition, the memories (430, 440) can store various information generated during the processing of one or more processors (410, 420). In addition, the memories (430, 440) can store various data and programs. The memories (430, 440) can include volatile memory (e.g., volatile memory (132) of FIG. 1) or nonvolatile memory (e.g., nonvolatile memory (134) of FIG. 1). The memories (430, 440) can store various data by having a large-capacity storage medium such as a hard disk.

[0150] Additionally, one or more processors (410, 420) may perform at least one method or a technique corresponding to at least one method related to the wearable electronic device (400) described through FIGS. 4A, 4B, 5, 6, 7A, 7B, 8, and 9. One or more processors (410, 420) may be electronic devices implemented as hardware having circuits having a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in a program. For example, a wearable electronic device (400) implemented in hardware may include a microprocessor, a central processing unit (CPU), a graphic processing unit (GPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), and / or a neural processing unit (NPU).

[0151] FIG. 5 is a diagram showing the arrangement of cameras and lights of a first tracking device and a second tracking device in a wearable electronic device according to one embodiment. Referring to FIG. 5, in a wearable electronic device (400) according to an embodiment (e.g., the wearable electronic device (200) of FIG. 2 and / or the wearable electronic device (300) of FIGS. 3A and 3B), the first cameras (e.g., the first primary camera (461), the first secondary camera (462)) of the first tracking device (e.g., the first tracking device (460) of FIG. 4B) (e.g., the second cameras (275a, 275b) of FIG. 2 and / or the third functional cameras (328a, 328b) of FIGS. 3A and 3B) may be positioned at positions (e.g., on the sides of each eye) capable of tracking the left and right eyes (or the gazes of the left and right eyes, respectively) of the user. The first cameras (461, 462) may recognize the irises of the left and right eyes of the user. At this time, the first lights (465, 466) may be arranged in multiple numbers around each eyeball to reflect multiple glint images to the user's left and right eyes, respectively. The first cameras (461, 462) may capture the glint images reflected to the user's left and right eyes by the first lights (465, 466).

[0152] In the wearable electronic device (400), the second cameras (e.g., the 2-1 secondary camera (481a), the 2-2 secondary camera (481b), and the 2-3 secondary camera (481c)) of the second tracking device (e.g., the 4th cameras (280a, 280b, 280c) of FIG. 2 and / or the 4th functional cameras (325, 326, 327) of FIGS. 3A and 3B) may be positioned at positions capable of recognizing a user's facial expression (e.g., a smiling expression, a frowning expression, a crying expression, or an angry expression). The second cameras (481a, 481b, 481c) may be positioned at positions capable of recognizing an expression of the lower part of the face of a user wearing the wearable electronic device (400) and / or a position capable of recognizing an expression between the eyebrows of the user. At this time, the second lights (485a, 485b, 485c) may be positioned adjacent to the second cameras (481a, 481b, 481c) to reflect infrared light onto the user's face, and enable the second cameras (481a, 481b, 481c) to capture the infrared light reflected from the face. The wearable electronic device (400) may analyze the captured user's face to recognize a facial expression, and then implement an avatar's expression corresponding to the recognized facial expression.

[0153] At this time, as indicated by the dotted line in FIG. 5, the two first lights (ET IR LED) (465) for the first primary camera (461) located at the bottom and the second light (485a) of the 2-1 secondary camera (481a) may be arranged close to each other, and some (e.g., two) first lights (ET IR LED) (466) for the 1st secondary camera (462) located at the bottom and the second light (485c) of the 2-3 secondary camera (481c) may be arranged close to each other. In addition, the second light (485b) for the 2-2 secondary camera (481b) may be arranged close to some of the first lights (ET IR LED) (465) and / or the first lights (ET IR LED) (466) located close to the user's forehead.

[0154] In a structure in which some lights are arranged in close proximity, for example, when the first light and the second light are turned on at the same time, or when the first lights (465, 466) and the second lights (485a, 485b, 485c) are fed into the second cameras (e.g., the 2-1 secondary camera (481a), the 2-2 secondary camera (481b), and the 2-3 secondary camera (481c)), a portion in which the brightness difference is large may occur in the images captured by the second cameras due to the influence of the first light.

[0155] In this specification, the phenomenon in which an image captured by a camera is interfered with (e.g., difference in brightness) by a different light (e.g., first light) for a different camera (e.g., first camera) than the light (e.g., second light) for the camera in question is called 'crosstalk'.

[0156] A wearable electronic device (400) according to one embodiment can prevent crosstalk caused by first and second lights by adjusting the exposure time for a first photographing device ('first exposure time') and the exposure time for a second photographing device ('second exposure time') so that they do not overlap. A method by which the wearable electronic device (400) adjusts the first exposure time and the second exposure time will be described in more detail with reference to FIG. 6 below.

[0157] FIG. 6 is a drawing (600) for explaining the temporal relationship between a video frame and lighting used in a wearable electronic device according to one embodiment. Referring to FIG. 6, a timing diagram (610) showing a first exposure time (Primary Exposure time) of a first primary camera (e.g., Primary-ET_L) (e.g., the first primary camera (461) of FIG. 4b) according to a signal (e.g., Fsync) indicating the start of a frame of a first primary camera (e.g., Primary-ET_L) (e.g., the first primary camera (461) of FIG. 4b) in a wearable electronic device according to an embodiment of the present invention, and a second exposure time (Secondary FT_R Exposure time) of a second secondary camera (e.g., FT_R) (e.g., the second secondary cameras (481) of FIG. 4b) according to a first signal (e.g., Strobe out) (621) related to the exposure of the first primary camera output from the first primary camera is shown. A timing diagram (620) and a timing diagram (630) showing the occurrence of a third signal (e.g., Strobe out) (635) related to the exposure of a second secondary camera (e.g., Secondary FT_R) are shown.

[0158] The 'signal indicating the start of a frame (Fsync)' may correspond to an I / O capable of controlling the start time of a frame. In addition, the 'signal related to exposure (Strobe)' of the camera may correspond to an I / O that notifies the wearable electronic device of the start time of the exposure (exposure) time of the camera. The wearable electronic device may, if necessary, generate the signal related to exposure (Strobe) of the camera before the start of the exposure time of the camera. In one embodiment, the signal indicating the start of a frame (Fsync) and the signal related to exposure (Strobe) of the camera may be used to prevent crosstalk from occurring between a first tracking device (e.g., a first tracking device (450) of FIG. 4A and / or a first tracking device (460) of FIG. 4B) and a second tracking device (e.g., a second tracking device (470) of FIG. 4A and / or a second tracking device (480) of FIG. 4B).

[0159] For example, when a signal (Fsync) indicating the start of a frame of the first primary camera (ET_L) is generated in the first tracking device (e.g., the first tracking device (450) of FIG. 4A and / or the first tracking device (460) of FIG. 4B), the first primary camera (ET_L) can capture and transmit image frames for 60 frames per second (fps) = 16.6 ms. At this time, the wearable electronic device can adjust the generation time of the first signal (Strobe out) (621) related to the exposure of the first primary camera (ET_L) so that the first exposure time of the first primary camera (ET_L) and the second exposure time of the second secondary camera (FT_R) of the second tracking device (e.g., the second tracking device (470) of FIG. 4A and / or the second tracking device (480) of FIG. 4A) do not overlap with each other.

[0160] The wearable electronic device may use the output (Strobe Output) of the first signal (621) related to the exposure of the first primary camera (ET_L) as a signal input (Fsync Input) notifying the start of a frame of the second tracking device (e.g., the second secondary camera (FT_R)) before the start of the first exposure time (615) of the first tracking device (e.g., the first primary camera (ET_L)), thereby causing a delay between the first exposure time of the first tracking device (e.g., the first primary camera (ET_L)) and the second exposure time of the second tracking device (e.g., the second secondary camera (FT_R)). In this case, the first exposure time and the second exposure time may not overlap each other due to the delay between the first exposure time of the first tracking device and the second exposure time of the second tracking device. At this time, the signal (Fsync) indicating the start of the frame of the second tracking device (e.g., the second secondary camera (FT_R)) may correspond to a signal to the second secondary camera (FT_R) to count 16.6 ms from now. The second secondary camera (FT_R) may perform an exposure for 4 ms according to a third signal (e.g., Strobe out) (635) that occurs 4 ms before (or slightly before) the end of 16.6 ms according to the count. While the second secondary camera (FT_R) is exposing, the second lights for the second secondary camera (FT_R) may also be turned on.

[0161] For example, the wearable electronic device can prevent crosstalk between the first tracking device and the second tracking device by adjusting the generation time of the first signal so that the second exposure time (FT CAM Exposure time) is less than the transmission time of one data frame transmitted by the first primary camera (1 frame time = 16.6 ms) minus the first exposure time (ET CAM Exposure time). At this time, the first tracking device and the second tracking device operate in the same cycle, but operate with a constant delay time (640) within the same cycle, so that the facial expression including the user's gaze can be reflected more accurately in the avatar system. At this time, the constant delay time (640) may be, for example, (the second exposure time (4 ms) + A ms) based on the start point of the exposure time of each camera of the first tracking device and the second tracking device. At this time, A ms may be controlled (or adjusted) by the first signal (e.g., Strobe out) (621).

[0162] According to the timing diagrams (610, 620) illustrated in FIG. 6, the first exposure time of the first primary camera may be, for example, 2 ms, and the second exposure time of the second secondary camera (e.g., FT_R) may be, for example, 4 ms, but is not necessarily limited thereto. The first exposure time of the first primary camera and the second exposure time of the second secondary camera may vary depending on the embodiment.

[0163] FIG. 7A is a diagram for explaining the input / output relationship between a primary camera and secondary cameras in a wearable electronic device according to one embodiment.

[0164] Referring to FIG. 7A, one of the first cameras (713, 716) (e.g., the second cameras (275a, 275b) of FIG. 2, the third functional cameras (328a, 328b) of FIGS. 3A and 3B, the first cameras (453) of FIG. 4A, and / or the first primary camera (461) and the second secondary camera (462) of FIG. 4B) of the first tracking device (710) for tracking the gaze of a user according to an embodiment of the present invention is set as a primary camera, and the other one of the first cameras and the second cameras of the second tracking device (730) for tracking the face of the user (e.g., the second tracking device (470) of FIG. 4A and / or the second tracking device (480) of FIG. 4B) are set as a primary camera. The input / output relationship of a wearable electronic device (700) set as a secondary camera (e.g., the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIGS. 3A and 3B, and / or the wearable electronic device (400) of FIGS. 4A and 4B) is illustrated.

[0165] A wearable electronic device (700) may include a first tracking device (710), a second tracking device (730), a first optical driver (750) (e.g., the first optical driver (463, 464) of FIG. 4B), and a second optical driver (770) (e.g., the second optical driver (483) of FIG. 4B).

[0166] The first tracking device (710) may include a left-eye camera (ET_Left) which is a first primary camera (713) (e.g., the first primary camera (461) of FIG. 4b) and a right-eye camera (ET_Right) which is a first secondary camera (716) (e.g., the first secondary camera (462) of FIG. 4b). The second tracking device (730) may include second secondary cameras (e.g., the 2-1 secondary camera (731), the 2-2 secondary camera (733), and the 2-3 secondary camera (736)) (e.g., the second cameras (473) of FIG. 4a) and / or the second secondary cameras (481) of FIG. 4b).

[0167] The first optical driver (750) can control the settings (e.g., intensity, direction, and / or on / off lighting) of the first lights (e.g., the first lights (451) of FIG. 4A and / or the first infrared lights (465, 466) of FIG. 4B) for the first cameras (713, 716) of the first tracking device (710) according to a control signal of a processor (e.g., the processor (120) of FIG. 1, the processor (410) of FIG. 4A, and / or the processor (420) of FIG. 4B).

[0168] The first optical driver (750) may include a 1-1 optical driver (ET IR LED Driver Left) (751) for left-eye illumination among the first infrared illuminations and a 1-2 optical driver (ET IR LED Driver Right) (753) for right-eye illumination among the first infrared illuminations.

[0169] The second optical driver (770) controls the second lights (e.g., the second lights (471) of FIG. 4a and / or the second infrared lights (485a, 485b, 485c) of FIG. 4b) for the cameras (731, 733, 736) of the second tracking device (730) (e.g., the fourth cameras (280a, 280b, 280c) of FIG. 2, the fourth functional cameras (325, 326, 327) of FIGS. 3a and 3b, the second cameras (473) of FIG. 4a and / or the second cameras (481a, 481b, 481c) of FIG. 4b) according to a control signal of the processor (e.g., the processor (120) of FIG. 1, the processor (410) of FIG. 4a, and / or the processor (420) of FIG. 4b). Settings (e.g., intensity, direction, and / or on / off lighting) can be adjusted. At this time, the settings of the cameras of the second tracking device (730) can also be adjusted according to a control signal from a processor (e.g., processor (120) of FIG. 1, processor (410) of FIG. 4A, and / or processor (420) of FIG. 4B).

[0170] According to one embodiment, whether the second lights for the cameras (731, 733, 736) of the second tracking device (730) are turned on / off, i.e., the synchronization of the second lights, can be controlled by a first signal transmitted by the first primary camera (713). The second light driver (770) may be located, for example, in a power management integrated circuit (PMIC) for a mobile application processor.

[0171] For example, the 2-1 secondary camera (731) may be a camera (FT_Left) that captures the left side of the user's face. The 2-2 secondary camera (733) may be a camera (FT_Middle (Brow)) that captures the center of the user's face (e.g., between the eyebrows). The 2-3 secondary camera (736) may be a camera (FT_Right) that captures the right side of the user's face. Not all of the 2nd secondary cameras (731, 733, 736) need to be used, and depending on the embodiment, the 2-1 secondary camera (731) and the 2-3 secondary camera (736) may be used, and the 2-2 secondary camera (733) may be selectively used.

[0172] The wearable electronic device (700) can input a signal (Fsync out) indicating the start of a frame output from the first primary camera (713) as a signal (Fsync in) indicating the start of a frame to the first secondary camera (716). At this time, since the first primary camera (713) and the first secondary camera (716) are synchronized with each other, even if a signal for the first primary camera (713) is used as a signal for the first secondary camera (716), a problem due to time synchronization between them may not occur.

[0173] The wearable electronic device (700) can input a first signal (Strobe out) related to the exposure of the first primary camera (713) as a signal (Fsync in) notifying the start of a frame to the second secondary cameras (731, 733, 736).

[0174] The wearable electronic device (700) can input a second signal (Strobe out) related to the exposure of the first secondary camera (716) as a first trigger signal (TRIG) corresponding to the first-1 optical driver (751) and the first-2 optical driver (753). As described below, the first-1 optical driver (751) and the first-2 optical driver (735) operate in a master and slave form and can communicate with an application processor (AP) via I2C (inter integrated circuit) communication.

[0175] In addition, the wearable electronic device (700) can input a third signal (Strobe out) related to the exposure of the second cameras (731, 733, 736) as a second trigger signal corresponding to the second optical driver (770). For example, the wearable electronic device (700) can input the third signals related to the exposure of each of the second-1 secondary camera (731) and the second-2 secondary camera (733) among the second cameras (731, 733, 736) as a second trigger signal corresponding to the second optical driver (770). At this time, the third signal related to the exposure of the second-1 secondary camera (731) can be input as a trigger signal for controlling the lights illuminating the left and right sides of the user's face among the second lights controlled by the second optical driver (770). The third signal related to the exposure of the 2-2 secondary camera (733) may be input as a trigger signal (Middle (Brow) TRIG) for controlling the lighting that illuminates the center of the user's face among the second lights controlled by the 2nd optical driver (770). The reason for separating the trigger signals in this way may be to prepare for a case where, depending on the embodiment, the 2-1 secondary camera (731) and the 2-3 secondary camera (736) among the second cameras are used, and the 2-2 secondary camera (733) is used selectively. According to one embodiment, the trigger signal for controlling the lighting that illuminates the left and right sides of the user's face among the second lights and the trigger signal (Middle (Brow) TRIG) for controlling the lighting that illuminates the center of the user's face among the second lights may be input as the third signal by one of the 2nd cameras (731, 733, 736) to the optical driver (770).In FIG. 7A, the 2-1 secondary camera (731) is shown as inputting the third signal as a trigger signal for controlling the lights illuminating the left and right sides of the user's face among the second lights controlled by the second optical driver (770), but the 2-3 secondary camera (736) may input the third signal as a trigger signal for controlling the lights illuminating the left and right sides of the user's face among the second lights controlled by the second optical driver (770).

[0176] FIG. 7b is a diagram for explaining the input / output relationship between the primary camera and the secondary cameras in a wearable electronic device according to one embodiment.

[0177] Referring to FIG. 7B, a wearable electronic device is provided in which one of three cameras (721, 723, 725) of a second tracking device (720) for tracking a user's face (e.g., the second tracking device (470) of FIG. 4A, the second tracking device (480) of FIG. 4B, and / or the second tracking device (730) of FIG. 7A) is set as a primary camera (e.g., the 2-1 camera (721)), and the remaining two cameras (e.g., the 2-2 camera (723) and the 2-3 camera (725)) and the cameras (741, 743) of a first tracking device (740) for tracking the user's gaze (e.g., the first tracking device (450) of FIG. 4A, the first tracking device (460) of FIG. 4B, and / or the first tracking device (710) of FIG. 7A) are set as secondary cameras. The input / output relationship of a device (701) (e.g., a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIGS. 3A and 3B, a wearable electronic device (400) of FIGS. 4A and 4B, and / or a wearable electronic device (700) of FIG. 7A) is illustrated.

[0178] The wearable electronic device (701) may include a second tracking device (720), a first tracking device (740), a second optical driver (760) (e.g., the second optical driver (483) of FIG. 4B and / or the second optical driver (770) of FIG. 7A), and a first optical driver (780) (e.g., the first optical drivers (463, 464) of FIG. 4B and / or the first optical driver (750) of FIG. 7A).

[0179] The second tracking device (720) may include a first primary camera (e.g., a 2-1 camera (721)) and first secondary cameras (e.g., a 2-2 camera (723) and a 2-3 camera (725)).

[0180] The first tracking device (740) may include second secondary cameras (e.g., a left eye camera (ET_Left) (741) and a right eye camera (ET_Right) (743)).

[0181] The second optical driver (760) may control the settings (e.g., intensity, direction, and / or on / off lighting) of the second lights (e.g., the second lights (471) of FIG. 4A and / or the second infrared lights (485a, 485b, 485c) of FIG. 4B) for the cameras (721, 723, 725) of the second tracking device (720) according to a control signal of a processor (e.g., the processor (120) of FIG. 1, the processor (410) of FIG. 4A, and / or the processor (420) of FIG. 4B). The second optical driver (760) may be located, for example, in a power management integrated circuit (PMIC) for a mobile application processor.

[0182] The second optical driver (760) can control the on / off of the second lights for the cameras (e.g., 721, 723, 725) of the second tracking device (720) according to the control signal of the second tracking device (720).

[0183] The first optical driver (780) may include a 1-1 optical driver (ET IR LED Driver Left) (781) for left-eye illumination among the first infrared lights (e.g., the first lights (451) of FIG. 4A and / or the first infrared lights (465, 466) of FIG. 4B) and a 1-2 optical driver (ET IR LED Driver Right) (783) for right-eye illumination among the first infrared lights.

[0184] For example, the 2-1 camera (721), which is the first primary camera, may be a camera (FT_Left) that captures the left side of the user's face. The 2-2 camera (723), which is the first secondary camera, may be a camera (FT_Right) that captures the right side of the user's face. The 2-3 camera (725), which is the first secondary camera, may be a camera (FT_Middle (Brow)) that captures the center of the user's face (e.g., between the eyebrows). Not all of the first secondary cameras (723, 725) need to be used, and depending on the embodiment, the 2-3 camera (725) may be used, and the 2-2 camera (723) may be selectively used.

[0185] In FIG. 7b, the wearable electronic device (701) can input a signal (Fsync out) indicating the start of a frame output from the 2-1 camera (721), which is the first primary camera, as a signal (Fsync in) indicating the start of a frame to the 2-2 camera (723) and the 2-3 camera (725), which are the first secondary cameras. At this time, since the 2-1 camera (721), which is the first primary camera, and the first secondary cameras (723, 725) are synchronized with each other, even if a signal for the 2-1 camera (721) is used as a signal for the 2-2 camera (723) or the 2-3 camera (725), which are the first secondary cameras, no problem due to time synchronization between them may occur.

[0186] The wearable electronic device (701) can input a first signal (Strobe out) related to the exposure of the first primary camera, the second-1 camera (721), and a signal (Fsync in) notifying the start of a frame to the second secondary cameras (741, 743).

[0187] The wearable electronic device (701) can input a second signal (Strobe out) related to the exposure of the second-second camera (723), which is the first secondary camera, as a second trigger signal corresponding to the second optical driver (760). For example, the wearable electronic device (701) can input a third signal related to the exposure of the second-second camera (723) among the second cameras (721, 723, 725) as a trigger signal (Left / Right TRIG) that controls the lighting that illuminates the left and right sides of the user's face among the second lights controlled by the second optical driver (760). The wearable electronic device (701) may input a signal related to the exposure of the second-third camera (725) among the second cameras (721, 723, 725) as a trigger signal (Middle (Brow) TRIG) that controls the lights illuminating the center of the user's face among the second lights controlled by the second light driver (760). In this way, the reason for separating the trigger signal may be to prepare for a case where the second-third camera (725) among the second cameras is selectively used, depending on the embodiment.

[0188] In addition, the wearable electronic device (701) can input a signal (Strobe out) related to the exposure of the first cameras (741, 743), which are second secondary cameras, as a first trigger signal corresponding to the first optical driver (780). For example, the wearable electronic device can input a signal (Strobe out) related to the exposure of the first camera (741) as a first trigger signal for each of the first-1 optical driver (781) and the first-2 optical driver (783). In FIG. 7B, the first camera (741) is illustrated as inputting the first trigger signal for each of the first-1 optical driver (781) and the first-2 optical driver (783), but the first camera (743) can input the first trigger signal for each of the first-1 optical driver (781) and the first-2 optical driver (783).

[0189] FIG. 8 is a diagram illustrating the operation between the optical drivers for the first lights of the first tracking device and the AP (application processor) in the wearable electronic device according to one embodiment.

[0190] Referring to FIG. 8, a drawing is shown showing a state in which a first signal (Strobe out) related to the exposure of the right-eye camera (ET_Right), which is a first secondary camera (716) according to one embodiment, is connected to a trigger input of a first optical driver (e.g., a 1-1 optical driver (751), a 1-2 optical driver (753)).

[0191] The 1-1 optical driver (751) and the 1-2 optical driver (753) operate in a master and slave form and can communicate with an application processor (AP) (810) (e.g., the processor (120) of FIG. 1, the processor (410) of FIG. 4A, and / or the processor (420) of FIG. 4B) via I2C (inter integrated circuit) communication. The I2C communication can be performed using a serial computer bus. The I2C communication uses two bidirectional open collector lines, serial data (SDA) and serial clock (SCL), to which pull-up resistors are connected. The master outputs a clock for synchronization with the serial clock (SCL), and the slave can perform both transmission and reception of serial data (SDA) in accordance with the clock output as the serial clock (SCL).

[0192] The AP (810) can control each of the first-first optical driver (751) and the first-second optical driver (753) according to the application. The AP (810) can transmit signals for controlling channel selection, such as how many lights among ten first lights are to be turned on and at which location, the amount of light of the turned-on lights, and / or the current value, to each of the first-first optical driver (751) and the first-second optical driver (753).

[0193] FIG. 9 is a diagram illustrating an example of an arrangement between a first tracking device, a second tracking device, and optical drivers in a wearable electronic device according to one embodiment. Referring to FIG. 9, a wearable electronic device (900) according to an embodiment (e.g., the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIGS. 3A and 3B, the wearable electronic device (400) of FIG. 4, the wearable electronic device (700) of FIG. 7A and / or the wearable electronic device (701) of FIG. 7B) includes a first secondary camera, a right-eye camera (ET_Right), and a first optical driver (920) for the first tracking device (910) (e.g., the first tracking device (450) of FIG. 4A, the first tracking device (460) of FIG. 4B, the first tracking device (710) of FIG. 7A, the first tracking device (740) of FIG. 7B) on the right side relative to the left-eye camera (ET_Left) of the first tracking device (910) used as a primary camera, and The first optical driver (463, 464) of FIG. 4b, the first optical driver (750) of FIG. 7a, and / or the first optical driver (780) of FIG. 7b) may be disposed, and on the left side, a second tracking device (930) (e.g., the second tracking device (470) of FIG. 4a, the second tracking device (480) of FIG. 4b, the second tracking device (730) of FIG. 7a, and / or the second tracking device (720) of FIG. 7b) and a second optical driver (940) for the second tracking device (930) (e.g., the second optical driver (483) of FIG. 4b, the second optical driver (770) of FIG. 7a, and / or the second optical driver (760)) may be disposed, but is not necessarily limited thereto.

[0194] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0195] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0196] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0197] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101) of FIG. 1). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0198] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0199] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0200] According to one embodiment, a wearable electronic device (200, 300, 400, 700, 701, 900) includes one or more processors (120, 410, 420), a memory (130, 430, 440) storing instructions to be executed by the processors (120, 410, 420), first lights (451, 465, 466) corresponding to a first area of ​​a user wearing the wearable electronic device (200, 300, 400, 700, 701, 900), and first cameras (275a, 275b, 328a, 328b, 453, 461, 462, 713, 716, 723, 725) corresponding to the first area. A first tracking device (450, 460, 710, 740, 910) including a first tracking device and a second tracking device (470, 480, 730, 720, 930) including second lights (471, 485a, 485b, 485c) corresponding to a second area of ​​the user and second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) corresponding to the second area, wherein the instructions, when executed by the processor (120, 410, 420), cause the processor (120, 410, 420) to: The first primary camera (461, 713, 721) among the first primary cameras (328a, 328b, 453, 461, 462, 713, 716, 723, 725) may be configured to execute an operation of controlling the first primary camera (461, 713, 721) to generate a first signal (621) related to the exposure of the first primary camera (461, 713, 721) and input the first signal (621) as a signal notifying the start of a frame to the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736).

[0201] According to one embodiment, the operation of inputting a signal indicating the start of the frame of the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) may be configured to execute an operation of controlling the first primary camera (461, 713, 721) to generate the first signal (621) at the first exposure time of the first primary camera (461, 713, 721) and input the first signal (621) as a signal indicating the start of the frame of the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736). there is.

[0202] According to one embodiment, the operation of inputting a signal indicating the start of the frame of the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) may include an operation of controlling the first primary camera (461, 713, 721) to adjust the generation time of the first signal (621) so that the first exposure time of the first primary camera (461, 713, 721) and the second exposure time of the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) do not overlap each other. there is.

[0203] According to one embodiment, the operation of controlling the time of occurrence of the first signal (621) may include an operation of controlling the first primary camera (461, 713, 721) to control the time of occurrence of the first signal (621) such that the interval between the first exposure time and the second exposure time is reduced in proportion to the time it takes the first primary camera (461, 713, 721) to transmit one data frame.

[0204] According to one embodiment, the operation of controlling the time of occurrence of the first signal (621) may include an operation of controlling the first primary camera (461, 713, 721) to control the time of occurrence of the first signal (621) so that the second exposure time is less than or equal to a value obtained by subtracting the first exposure time from the transmission time of one data frame transmitted by the first primary camera (461, 713, 721).

[0205] According to one embodiment, the first area corresponds to the left eye of the user and the right eye of the user, and the first tracking device (450, 460, 710, 740, 910) may include the first lights (451, 465, 466) that generate images reflected in the left eye and the right eye of the user wearing the wearable electronic device (200, 300, 400, 700, 701, 900), a first primary camera (461, 713, 721) that tracks the images reflected in the left eye of the user and the left eye of the user, and a first secondary camera (462, 716, 723, 725) that tracks the images reflected in the right eye of the user and the right eye of the user.

[0206] According to one embodiment, the second area corresponds to the user's face, and the second tracking device (470, 480, 730, 720, 930) may include second lights (471, 485a, 485b, 485c) that reflect light onto the user's face, and second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) that recognize the user's facial expression by the second lights (471, 485a, 485b, 485c).

[0207] According to one embodiment, the first cameras (275a, 275b, 328a, 328b, 453, 461, 462, 713, 716, 723, 725) can be synchronized with each other.

[0208] According to one embodiment, the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) can be synchronized with each other.

[0209] According to one embodiment, the first tracking device (450, 460, 710, 740, 910) and the second tracking device (470, 480, 730, 720, 930) can operate with a constant delay within the same cycle.

[0210] According to one embodiment, the instructions, when executed by the processor (120, 410, 420), may be configured to cause the processor (120, 410, 420) to perform an operation of inputting a signal indicating the start of a frame output from the first primary camera (461, 713, 721) as a signal indicating the start of the frame to the first secondary camera (462, 716, 723, 725) among the first cameras (275a, 275b, 328a, 328b, 453, 461, 462, 713, 716, 723, 725).

[0211] According to one embodiment, the instructions, when executed by the processor (120, 410, 420), may be further configured to cause the processor (120, 410, 420) to input a second signal related to exposure of a first secondary camera (462, 716, 723, 725) of the first camera as a first trigger signal corresponding to a first-1 light driver for left-eye lights among the first lights (451, 465, 466) and a first-2 light driver for right-eye lights among the first lights (451, 465, 466).

[0212] According to one embodiment, the first-1 optical driver and the first-2 optical driver operate in a master and slave form and can communicate with the processor (120, 410, 420) through I2C (Inter Integrated Circuit) communication.

[0213] According to one embodiment, the instructions, when executed by the processor (120, 410, 420), may be further configured to cause the processor (120, 410, 420) to input a third signal (635) related to exposure of the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) as a second trigger signal corresponding to a second light driver (483, 770, 760) for the second lights (471, 485a, 485b, 485c).

[0214] According to one embodiment, the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) include a 2-1 secondary camera (731), a 2-2 secondary camera (733), and a 2-3 secondary camera (736), wherein the 2-1 secondary camera (731) can capture a left side of the user's face, the 2-2 secondary camera (733) can capture a center part of the user's face, and the 2-3 secondary camera (736) can capture a right side of the user's face.

[0215] According to one embodiment, the instructions, when executed by the processor (120, 410, 420), may be configured to further cause the processor (120, 410, 420) to input third signals (635) related to the exposure of each of the second-first secondary camera (731) and the second-second secondary camera (733) among the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) as second trigger signals corresponding to the second light drivers (483, 770, 760) for the second lights (471, 485a, 485b, 485c).

[0216] According to one embodiment, a wearable electronic device (200, 300, 400, 700, 701, 900) includes a first tracking device (450, 450) including one or more processors (120, 410, 420), a memory (130, 430, 440) storing instructions to be executed by the processors (120, 410, 420), first infrared lights generating images reflected to the left and right eyes of a user wearing the wearable electronic device (200, 300, 400, 700, 701, 900), and a first primary camera (461, 713, 721) tracking the reflected images and the left eye and a first secondary camera (462, 716, 723, 725) tracking the right eye. A second tracking device (470, 480, 730, 720, 930) comprising second infrared lights reflecting light onto the user's face, and second secondary cameras (481, 731, 733, 736, 741, 743) recognizing the user's facial expression by the second infrared lights, wherein the instructions, when executed by the processor (120, 410, 420), cause the processor (120, 410, 420) to transmit a first signal (621) related to the exposure of the first primary camera (461, 713, 721) to the first primary camera (461, 713, 721). 721) can be configured to execute an operation of controlling the input of a signal to notify the start of a frame to the second secondary cameras (481, 731, 733, 736, 741, 743).

[0217] According to one embodiment, the instructions, when executed by the processor (120, 410, 420), may control the processor (120, 410, 420) to adjust the generation time of the first signal (621) such that the first exposure time of the first primary camera (461, 713, 721) and the second exposure time of the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) do not overlap with each other.

[0218] According to one embodiment, the second cameras (280a, 280b, 280c, 325, 326, 327, 473, 721, 723, 725, 731, 733, 736) include a 2-1 secondary camera (731), a 2-2 secondary camera (733), and a 2-3 secondary camera (736), wherein the 2-1 secondary camera (731) can capture a left side of the user's face, the 2-2 secondary camera (733) can capture a center part of the user's face, and the 2-3 secondary camera (736) can capture a right side of the user's face.

[0219] According to one embodiment, the wearable electronic device (200, 300, 400, 700, 701, 900) further includes at least one of a first-first optical driver for controlling left-eye illuminations among the first infrared illuminations according to a second signal related to exposure of the first secondary camera (462, 716, 723, 725), a first-second optical driver for controlling right-eye illuminations among the first infrared illuminations according to the second signal, and a second optical driver (483, 770, 760) for controlling second infrared illuminations according to a third signal (635) related to exposure of the second secondary cameras (481, 731, 733, 736, 741, 743). 701, 900).

Claims

1. In wearable electronic devices, One or more processors; A memory that stores instructions to be executed by the processor; A first tracking device including first lights corresponding to a first area of ​​a user wearing the wearable electronic device and first cameras corresponding to the first area; and A second tracking device comprising second lights corresponding to the second area of ​​the user and second cameras corresponding to the second area. Including, The above memory When executed by said one or more processors, said wearable electronic device causes: An operation of controlling a first primary camera among the first cameras to generate a first signal related to the exposure of the first primary camera and input the first signal as a signal notifying the start of a frame to the second cameras. A wearable electronic device storing one or more computer programs comprising computer-executable instructions causing the device to execute the instructions.

2. In paragraph 1, The operation of inputting the first signal as a signal to the second cameras to notify the start of the frame An operation in which the first primary camera generates the first signal at the first exposure time of the first primary camera to input the signal as an indication of the start of the frame of the second cameras. A wearable electronic device storing one or more computer programs comprising computer-executable instructions causing the device to execute the instructions.

3. In paragraph 2, The action of inputting a signal to the above second cameras to notify the start of the above frame is An operation for controlling the first primary camera to adjust the generation time of the first signal so that the first exposure time of the first primary camera and the second exposure times of the second cameras do not overlap each other. A wearable electronic device storing one or more computer programs comprising computer-executable instructions causing the device to execute the instructions.

4. In paragraph 3, The operation for controlling the occurrence time of the above first signal is An operation for controlling the first primary camera to adjust the timing of occurrence of the first signal so that the interval between the first exposure time and the second exposure time is reduced in proportion to the time it takes the first primary camera to transmit one data frame. A wearable electronic device storing one or more computer programs comprising computer-executable instructions causing the device to execute the instructions.

5. In paragraph 4, The operation for controlling the occurrence time of the above first signal is An operation for controlling the first primary camera to adjust the generation time of the first signal so that the second exposure time is less than or equal to a value obtained by subtracting the first exposure time from the transmission time of one data frame transmitted by the first primary camera. A wearable electronic device storing one or more computer programs comprising computer-executable instructions causing the device to execute the instructions.

6. In paragraph 1, The above first area corresponds to the user's left eye and the user's right eye, The above first tracking device The first lights generating images reflected into the left eye and the right eye of a user wearing the wearable electronic device; A first primary camera that tracks the images reflected on the left eye of the user and the left eye of the user; and Images reflected on the user's right eye and a first secondary camera tracking the user's right eye Including, The above first tracking device and the above second tracking device A wearable electronic device that operates with a constant delay within the same cycle.

7. In paragraph 1, The second region corresponds to the user's face, The second tracking device above said second lights reflecting light onto the face of said user; and The second cameras that recognize the user's facial expressions by the second lights Including, The above first tracking device and the above second tracking device A wearable electronic device that operates with a constant delay within the same cycle.

8. In paragraph 1, The above first cameras are synchronized with each other, The above second cameras are synchronized with each other, The above second cameras Including a 2-1 secondary camera, a 2-2 secondary camera, and a 2-3 secondary camera, The 2-1 secondary camera captures the left side of the user's face, the 2-2 secondary camera captures the center of the user's face, and the 2-3 secondary camera captures the right side of the user's face. The above instructions, when executed by the one or more processors, cause the wearable electronic device to: An operation of inputting third signals related to the exposure of each of the second cameras, the second-1 secondary camera and the second-2 secondary camera, as a second trigger signal corresponding to the second light driver for the second lights. A wearable electronic device storing one or more computer programs comprising computer-executable instructions that cause the device to execute further instructions.

9. In paragraph 1, The above instructions, when executed by the one or more processors, cause the wearable electronic device to: An operation of inputting a signal indicating the start of a frame output from the first primary camera as a signal indicating the start of the frame to the first secondary camera among the first cameras. A wearable electronic device storing one or more computer programs comprising computer-executable instructions that cause the device to execute further instructions.

10. In paragraph 1, The above instructions, when executed by the one or more processors, cause the wearable electronic device to: An operation of inputting a second signal related to the exposure of a first secondary camera among the first cameras as a first trigger signal corresponding to a first-1 optical driver for left-eye illumination among the first lights and a first-2 optical driver for right-eye illumination among the first lights storing one or more computer programs containing computer-executable instructions that cause the computer to execute further instructions; The above 1-1 optical driver and the above 1-2 optical driver A wearable electronic device that operates in master and slave mode and communicates with one or more processors via I2C (inter integrated circuit) communication.

11. In paragraph 1, The above instructions, when executed by the one or more processors, cause the wearable electronic device to: An operation of inputting a third signal related to the exposure of the second cameras as a second trigger signal corresponding to the second light driver for the second lights. A wearable electronic device storing one or more computer programs comprising computer-executable instructions that cause the device to execute further instructions.

12. In wearable electronic devices, One or more processors; A memory communicatively coupled to said one or more processors; A first tracking device including first infrared lights that generate images reflected to the left and right eyes of a user wearing the wearable electronic device, and a first primary camera that tracks the reflected images and the left eye, and a first secondary camera that tracks the images reflected to the right eye of the user and the right eye of the user; and A second tracking device comprising second infrared lights that reflect light onto the user's face, and second secondary cameras that recognize the user's facial expression by the second infrared lights. Including, The above memory A wearable electronic device storing one or more computer programs including computer-executable instructions that, when executed by said one or more processors, cause the wearable electronic device to perform an operation of causing the first primary camera to generate a first signal related to the exposure of the first primary camera at a first exposure time of the first primary camera, thereby inputting a signal to the second secondary cameras to notify the start of a frame.

13. In paragraph 12, The above memory When executed by said one or more processors, said wearable electronic device causes said A wearable electronic device storing one or more computer programs including computer-executable instructions that cause the first primary camera to execute an operation of controlling the timing of generation of the first signal so that the first exposure time of the first primary camera and the second exposure times of the second cameras do not overlap each other.

14. In paragraph 17, The above second secondary cameras Including a 2-1 secondary camera, a 2-2 secondary camera, and a 2-3 secondary camera, The above 2-1 secondary camera captures the left side of the user's face, The above 2-2 secondary camera captures the center of the user's face, A wearable electronic device in which the second to third secondary cameras photograph the right side of the user's face.

15. In paragraph 12, The above wearable electronic device A first-first optical driver for controlling left-eye lights among the first infrared lights according to a second signal related to the exposure of the first secondary camera; A first-second optical driver controlling right-eye lights among the first infrared lights according to the second signal; and A second optical driver controlling the second infrared lights according to a third signal related to the exposure of the second secondary cameras. A wearable electronic device comprising at least one of:

Citation Information

Patent Citations

  • Gaze tracking apparatus and gaze tracking method

    KR1020170091467A

  • Facial expression recognition system, facial expression recognition method, and facial expression recognition program

    KR1020180102591A

  • Wedge block for restrain apparatus and constructing method using the same

    KR102149489B1

  • Real-time video content streaming system and method

    KR102881526B1

  • Eye tracking module and head-wearable device

    US20220342224A1