Wearable device and method for identifying location of target object

Abstract

This wearable device comprises a display, one or more cameras, and at least one processor. The at least one processor is configured to identify information about a target object. The at least one processor is configured to identify, in at least one image, a visual object related to an external object corresponding to the target object. The at least one processor is configured to identify whether a first image including the visual object is displayed. The at least one processor is configured to modify the first image to emphasize the visual object on the basis of identifying that the first image including the visual object is displayed. The at least one processor is configured to, on the basis of identifying that a second image distinguished from the first image including the visual object is displayed, overlay an affordance for changing a user's gaze to display the first image, on to the second image.

Description

Wearable device and method for identifying the location of a target object

[0001] The descriptions below relate to a wearable device and method for identifying the location of a target object.

[0002] To provide an enhanced user experience, electronic devices that provide augmented reality (AR) and / or virtual reality (VR) services can display computer-generated information in conjunction with external objects in the real world. The electronic devices may be wearable devices that can be worn by a user. For example, the electronic devices may be AR glasses and / or a head-mounted device (HMD).

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] According to one embodiment, a wearable device may include a display, one or more cameras, a memory including one or more storage media storing instructions, and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify information about a target object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a visual object regarding an external object corresponding to the target object within at least one image acquired by the one or more cameras and based on a gaze of a user of the wearable device during a designated time period. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether a first image including the visual object is displayed on the display based on the gaze of the user. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to alter the first image to emphasize the visual object based on identifying that the first image including the visual object is displayed on the display.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display an affordance for shifting a user's gaze to display the first image superimposed on the second image based on identifying that a second image distinct from the first image including the visual object is displayed through the display.

[0005] According to one embodiment, a method of a wearable device may include an operation of identifying information about a target object, and an operation of identifying a visual object about an external object corresponding to the target object within at least one image acquired based on a gaze of a user of the wearable device during a specified time period. The method may include an operation of identifying, based on the gaze of the user, whether a first image including the visual object is displayed through a display of the wearable device. The method may include an operation of modifying the first image to emphasize the visual object based on identifying that the first image including the visual object is displayed through the display. The method may include an operation of superimposing an affordance on the second image to modify the gaze of the user to display the first image based on identifying that a second image distinct from the first image including the visual object is displayed through the display.

[0006] According to one embodiment, a method of a wearable device may include an operation of identifying information about a target object and an operation of requesting an external wearable device connected to the wearable device to identify a visual object related to the target object. The method may include an operation of receiving information about the identified visual object through the external wearable device based on the request and an operation of displaying an element for indicating a location of the external object based on the information about the visual object on a display of the wearable device.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0008] FIGS. 2A and 2B illustrate an example of a perspective view of a wearable device according to one embodiment.

[0009] FIGS. 3A and 3B illustrate an example of an appearance of a wearable device according to one embodiment.

[0010] FIG. 4 illustrates an example block diagram of a wearable device according to one embodiment.

[0011] FIG. 5 illustrates an example of a screen displayed through a display of a wearable device according to one embodiment.

[0012] FIG. 6 illustrates components of a wearable device for identifying an external object corresponding to a target object, according to one embodiment.

[0013] FIG. 7 illustrates a flowchart of the operation of a wearable device according to one embodiment.

[0014] FIG. 8A illustrates an example of operation of a wearable device according to one embodiment.

[0015] FIG. 8b illustrates an example of operation of a wearable device according to one embodiment.

[0016] FIG. 9 illustrates an example of operation of a wearable device according to one embodiment.

[0017] FIG. 10 illustrates an example of operation of a wearable device according to one embodiment.

[0018] FIG. 11 illustrates a flowchart of the operation of a wearable device according to one embodiment.

[0019] FIG. 12A illustrates an example of an operation of a wearable device to identify an external object corresponding to a target object through an external wearable device, according to one embodiment.

[0020] FIG. 12b illustrates an example of an operation of a wearable device to identify an external object corresponding to a target object through an external wearable device, according to one embodiment.

[0021] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0022] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0023] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0024] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0025] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0026] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0027] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0028] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0029] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0030] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0031] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0032] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0033] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0034] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0036] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0037] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0038] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0040] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0041] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0042] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0043] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0044] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101).

[0045] The electronic device (101) may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199). The electronic device (101) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] FIGS. 2A and 2B illustrate an example of a perspective view of a wearable device according to one embodiment.

[0047] According to one embodiment, the wearable device (200) may have the form of glasses that are wearable on a body part of a user (e.g., head). The wearable device (200) of FIGS. 2A and 2B may be an example of the electronic device (101) of FIG. 1. The wearable device (200) may include a head-mounted display (HMD). For example, the housing of the wearable device (200) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (200) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0048] Referring to FIG. 2A, according to one embodiment, a wearable device (200) may include at least one display (250) and a frame (295) supporting at least one display (250).

[0049] According to one embodiment, the wearable device (200) can be worn on a part of a user's body. The wearable device (200) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (200). For example, the wearable device (200) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0050] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0051] Referring to FIG. 2B, at least one display (250) can provide visual information transmitted from external light to the user through a lens included in the at least one display (250), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When a user wears the wearable device (200), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0052] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (200) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0053] The wearable device (200) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (200) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (200) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (200) can view an image displayed on at least one display (250).

[0054] According to one embodiment, the frame (295) may be formed as a physical structure that allows the wearable device (200) to be worn on the user's body. According to one embodiment, the frame (295) may be configured so that, when the user wears the wearable device (200), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (295) may support at least one display (250). For example, the frame (295) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0055] Referring to FIG. 2A, the frame (295) may include a region (220) that at least partially contacts a part of the user's body when the user wears the wearable device (200). For example, the region (220) of the frame (295) that contacts a part of the user's body may include a region that contacts a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (200) makes contact with. According to one embodiment, the frame (295) may include a nose pad (210) that contacts a part of the user's body. When the wearable device (200) is worn by the user, the nose pad (210) may contact a part of the user's nose. The frame (295) may include a first temple (204) and a second temple (205) that contact a part of the user's body that is distinct from the part of the user's body.

[0056] For example, the frame (295) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of an ear of the wearer, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of an ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (200) can identify an external object (e.g., a user's fingertip) touching the frame (295) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (295).

[0057] According to one embodiment, the wearable device (200) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within a frame (295).

[0058] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (200) may be disposed on at least a portion of the frame (295) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B , but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B . When the number of microphones (265) included in the wearable device (200) is two or more, the wearable device (200) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (295).

[0059] According to one embodiment, at least one optical device (282, 284) can project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) can be a projector. At least one optical device (282, 284) can be disposed adjacent to at least one display (250) or can be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (200) can include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0060] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (295) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (200). For example, the wearable device (200) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (200) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (200) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (200) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (200) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other. The wearable device (200) can obtain an image having visual quality of a first area matching the user's gaze and visual quality of a second area using foveated rendering.For example, if the wearable device (200) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1). An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0061] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (260-4) is superimposed on a virtual image provided through at least one optical device (282, 284). The wearable device (200) can compensate for depth information (e.g., the distance between the wearable device (200) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (260-4). The wearable device (200) can perform object recognition through an image acquired using a photographing camera (260-4). The wearable device (200) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the photographing camera (260-4). The wearable device (200) can perform a pass-through function to display an image acquired through the photographing camera (260-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the photographing camera (260-4) can be disposed on a bridge (203) disposed between a first rim (201) and a second rim (202).

[0062] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (200) and matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (200) looks straight ahead, the wearable device (200) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (200) is positioned.

[0063] The gesture recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The gesture recognition camera (260-2, 260-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using simultaneous localization and mapping (SLAM) and / or depth map for 6 degrees of freedom pose (6 dof pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) may be positioned on the first rim (201) and / or the second rim (202).

[0064] The camera (260) included in the wearable device (200) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (200) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (200) can identify an external object based on a sensor for identifying the distance between the wearable device (200) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (200) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (200).

[0065] Although not shown, in one embodiment, the wearable device (200) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (295) and the hinge units (206, 207).

[0066] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (200). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0067] The antenna module (275) can transmit signals or power to the outside of the wearable device (200), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0068] The speaker (255) can output an audio signal to the outside of the wearable device (200). The audio output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (200). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus adjacent to the user's right ear.

[0069] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (200) to the user. For example, when the wearable device (200) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0070] Referring to FIG. 2B, according to one embodiment, a wearable device (200) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (200) (e.g., hardwares illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (200) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0071] According to one embodiment, a wearable device (200) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (200) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (200) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (200) based on the IMU.

[0072] Figures 3a and 3b illustrate an example of an exterior appearance of a wearable device according to one embodiment. The wearable device (200) of Figures 3a and 3b may be an example of the electronic device (101) of Figure 1 and the wearable device (200) of Figures 2a and 2b. An example of an exterior appearance of a first side (310) of a housing of the wearable device (200) according to one embodiment is illustrated in Figure 3a, and an example of an exterior appearance of a second side (320) opposite to the first side (310) may be illustrated in Figure 3b.

[0073] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (200) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (200) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (200) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0074] According to one embodiment, the wearable device (200) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (200) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (200) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (200) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar representing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (200).

[0075] Referring to FIG. 3B, a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor 330) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A to obtain information related to the external environment of the wearable device (200). For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may refer to the motion recognition cameras (260-2, 260-3) of FIG. 2b.

[0076] For example, using cameras (260-11, 260-12), the wearable device (200) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be placed on the second face (320) of the wearable device (200) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be placed on the second face (320) of the wearable device (200) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can refer to the shooting camera (260-4) of FIG. 2B.

[0077] According to one embodiment, the wearable device (200) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (200) and an external object. Using the depth sensor (330), the wearable device (200) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (200). Although not illustrated, a microphone may be disposed on the second face (320) of the wearable device (200) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0078] FIG. 4 illustrates an example block diagram of a wearable device according to one embodiment. Referring to FIG. 4, a wearable device (200) according to one embodiment may include at least one of a processor (410), a memory (415), a display (420), a camera (425), a sensor (430), or a communication circuit (435). The processor (410), the memory (415), the display (420), the camera (425), the sensor (430), and the communication circuit (435) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (402). The type and / or number of hardware components included in the wearable device (200) are not limited to those illustrated in FIG. 4. For example, the wearable device (200) may include only some of the hardware components illustrated in FIG. 4. The elements within the memory described below (e.g., layers and / or modules) may be logically distinct, but are not limited thereto.

[0079] According to one embodiment, the processor (410) of the wearable device (200) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (410) may be one or more. For example, the processor (410) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0080] The memory (415) of the wearable device (200) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (410). The memory (415) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi media card (eMMC).

[0081] In one embodiment, the display (420) of the wearable device (200) can output visualized information to a user of the wearable device (200). For example, the display (420) can be controlled by a processor (410) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (420) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0082] In one embodiment, the camera (425) of the wearable device (200) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (425) may be arranged in the form of a two-dimensional array. The camera (425) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (425) may mean one (a) two-dimensional frame data acquired from the camera (425). For example, video data captured using the camera (425) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (425) according to a frame rate. The camera (425) may further include a flash light that is positioned toward the direction in which the camera (425) receives light and outputs light toward the direction.

[0083] According to one embodiment, the wearable device (200) may include a plurality of cameras, for example, a camera (425), arranged in different directions. The camera (425) may include one or more cameras (260) described previously. A first camera among the plurality of cameras may be referred to as a motion recognition camera (e.g., motion recognition cameras 260-2 and 260-3 of FIG. 2B ), and a second camera may be referred to as a gaze tracking camera (e.g., gaze tracking camera 260-1 of FIG. 2B ). The wearable device (200) may identify a position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (200) may identify a direction of a gaze of a user wearing the wearable device (200) using an image acquired using the second camera. For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0084] According to one embodiment, a sensor (430) of a wearable device (200) may generate electrical information that may be processed by a processor (410) and / or a memory (415) of the wearable device (200) from non-electronic information related to the wearable device (200). The information may be referred to as sensor data. The sensor (430) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (200), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (200).

[0085] In one embodiment, the communication circuit (435) of the wearable device (200) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (200) and an external electronic device. The communication circuit (435) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (435) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0086] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by the processor (410) of the wearable device (200) may be stored in the memory (415) of the wearable device (200). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (200) and / or the processor (410) may perform at least one of the operations according to the embodiments described below when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (200) may mean that one or more instructions provided in the form of an application are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (200)). As an example, an application may include a program and / or a library related to a service provided to a user.

[0087] Referring to FIG. 4, programs installed in the wearable device (200) may be classified into one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (200) (e.g., the display (420), the camera (425), and / or the sensor (430)) may be classified within the hardware abstraction layer (480). The framework layer (450) may be referred to as an XR framework layer in that it includes one or more programs for providing an XR (extended reality) service. For example, FIG. 4 illustrates layers being divided within the memory (415), the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (415).

[0088] For example, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a position tracker (471), a spatial recognizer (472) (the spatial recognizer (472) is also referred to as a spatial recognizer (427)), a gesture tracker (473), and / or an eye-gaze tracker (474), a face tracker (475)) may be classified. Programs classified within the framework layer (450) may provide an executable API (application programming interface) based on other programs.

[0089] For example, within the application layer (440), programs designed to target users controlling wearable devices (200) may be classified. Examples of programs classified into the application layer (440) include, but are not limited to, an XR (extended reality) system UI (user interface) and / or an XR application (442). For example, programs (e.g., software applications) classified into the application layer (440) may call an API (application programming interface) to cause execution of functions supported by programs classified into the framework layer (450).

[0090] For example, the wearable device (200) may display one or more visual objects on the display (420) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (441). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (200) may provide a service for controlling functions available within a virtual space to the user based on the execution of the XR system UI (441).

[0091] Referring to FIG. 4, the XR system UI (441) is illustrated to include, but is not limited to, a lightweight renderer (443) and / or an XR plug-in (444). For example, the XR system UI (441) may cause execution of functions supported by the lightweight renderer (443) and / or the XR plug-in (444).

[0092] For example, the wearable device (200) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (200) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (444). The XR plugin (444) may be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0093] For example, the wearable device (200) may display a screen representing at least a portion of a virtual space on the display (420) based on the execution of the XR application (442). The XR plug-in (444-1) included in the XR application (442) may be referenced by the XR plug-in (444) of the XR system UI (441). Descriptions of the XR plug-in (444-1) that overlap with the description of the XR plug-in (444) may be omitted. The wearable device (200) may cause the execution of the virtual space manager (451) based on the execution of the XR application (442).

[0094] According to one embodiment, the wearable device (200) may provide a virtual space service based on the execution of the virtual space manager (451). For example, the virtual space manager (451) may include a platform (e.g., an Android platform) for supporting the virtual space service. The wearable device (200) may display the posture of a virtual object representing the user's posture rendered using data acquired through the sensor (430) based on the execution of the virtual space manager (451) on the display. The virtual space manager (451) may be referred to as a composition presentation manager (CPM).

[0095] For example, the virtual space manager (451) may include a runtime service (452). As an example, the runtime service (452) may be referred to as an OpenXR runtime module. The wearable device (200) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (200) based on the execution of the runtime service (452). As an example, the wearable device (200) may be used to perform rendering for a virtual space service to a user based on the execution of the runtime service (452). For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the execution of the runtime service (452).

[0096] For example, the virtual space manager (451) may include a pass-through manager (453). Based on the execution of the pass-through manager (453), the wearable device (200) may display a screen representing a virtual space on the display (420), while another screen representing an actual space acquired through the camera (425) may be superimposed on at least a portion of the screen.

[0097] For example, the virtual space manager (451) may include an input manager (454). Based on the execution of the input manager (454), the wearable device (200) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (470). The wearable device (200) may initiate execution of at least one of the functions of the wearable device (200) using the acquired data.

[0098] For example, the perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). From the perspective of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referenced as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0099] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data acquired from a sensor (430) (or a camera (425)). The one or more programs may include at least one of a position tracker (471), a space recognizer (472), a gesture tracker (473), an eye tracker (474), and / or a face tracker (475). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those illustrated in FIG. 4.

[0100] For example, the wearable device (200) can identify the pose of the wearable device (200) using the sensor (430) based on the execution of the position tracker (471). The wearable device (200) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (200) using data acquired using the camera (425) and the IMU based on the execution of the position tracker (471). The position tracker (471) can be referred to as a head tracking (HeT) module.

[0101] For example, the wearable device (200) may be used to construct a three-dimensional virtual space around the wearable device (200) (or a user of the wearable device (200)) based on the execution of the space recognizer (472). The wearable device (200) may reconstruct the three-dimensional surroundings of the wearable device (200) using data acquired using the camera (425) based on the execution of the space recognizer (472). The wearable device (200) may identify at least one of a plane, a slope, and stairs based on the three-dimensionally reconstructed surroundings of the wearable device (200) based on the execution of the space recognizer (472). The space recognizer (472) may be referred to as a scene understanding (SU) module.

[0102] For example, the wearable device (200) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (200) based on the execution of the gesture tracker (473). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (430) based on the execution of the gesture tracker (473). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using a camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0103] For example, the wearable device (200) may identify (or track) eye movements of a user of the wearable device (200) based on the execution of the gaze tracker (474). As an example, the wearable device (200) may identify eye movements of the user using data acquired from at least one sensor based on the execution of the gaze tracker (474). As an example, the wearable device (200) may identify eye movements of the user based on data acquired using a camera (e.g., the gaze tracking camera (260-1) of FIGS. 2A and 2B) and / or an infrared light emitting diode (IR LED) based on the execution of the gaze tracker (474). The gaze tracker (474) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0104] For example, the recognition service layer (470) of the wearable device (200) may further include a face tracker (475) for tracking the user's face. For example, the wearable device (200) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (475). The wearable device (200) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (475). As an example, the wearable device (200) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (475).

[0105] For the embodiments described below, the wearable device (200) of FIG. 4 may be referred to. For example, the embodiments described below may be performed by the processor (410) of the wearable device (200) of FIG. 4.

[0106] FIG. 5 illustrates an example of a screen displayed through a display of a wearable device according to one embodiment.

[0107] Referring to FIG. 5, the wearable device (200) may correspond to the wearable device (200) illustrated in FIG. 4. For example, the wearable device (200) may include a camera (425) positioned toward the front of the user (510) when the device is worn by the user (510). The front of the user (510) may correspond to the direction in which the user's (510) head and / or the user's (510) gaze is directed. According to one embodiment, the wearable device (200) may include a sensor (430) for identifying the motion of the user's (510) head and / or the wearable device (200) when the device is worn by the user (510).

[0108] The processor (410) of the wearable device (200) can identify the angle of the wearable device (200) based on data acquired through the sensor (430). In order to provide a user interface (UI) based on virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) to a user (510) wearing the wearable device (200), the processor (410) can control the camera (425) and / or the sensor (430). The UI can be related to the wearable device (200), a metaverse service provided by a server connected to the wearable device (200), and / or a notification service.

[0109] According to one embodiment, the processor (410) may provide a screen (or virtual space) related to augmented reality (AR) and / or mixed reality (MR).

[0110] For example, the wearable device (200) may operate based on optical seethrough (OST). For example, the processor (410) may display a screen (e.g., screen (550)) (or image) including a mixed image of a real object and a virtual object) through the display (420). For example, while the user (510) is wearing the wearable device (200), the processor (410) may identify ambient light passing through at least one lens positioned adjacent to the eyes of the user (510). The ambient light passing through the lens may be combined (or mixed) with light emitted from the display (420) of the wearable device (200). The display area of ​​the display (420) may be formed within the lens through which the ambient light passes. Since the processor (410) combines the ambient light and the light emitted from the display (420), the user (510) can see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display (420).

[0111] For example, the wearable device (200) may operate based on VST (video seethrough). For example, the processor (410) may display a screen provided to the user (510) based on a video (or rendered image). For example, the processor (410) may obtain an image of the external environment (e.g., the external environment (501)) using the camera (425). The processor (410) may display a screen including an image of the external environment and a visual object overlapping the image through the display (420).

[0112] For example, the external environment (500) may include an external environment (501) and an external environment (502). The wearable device (200) may be positioned toward the external environment (501). The processor (410) may display a screen (550) corresponding to the external environment (501). If the wearable device (200) operates based on OST, the screen (550) may be configured to include real objects and virtual objects. If the wearable device (200) operates based on VST, the screen (550) may be configured to include an image acquired through the camera (425) and a visual object superimposed on the image.

[0113] According to one embodiment, the processor (410) may recognize a space (e.g., a real space) in which the user (510) (or the wearable device (200)) is located, and generate a virtual space mapped to the space. The spatial recognition performed by the processor (410) may include simultaneous localization and mapping (SLAM) and / or spatial mapping (e.g., scene understanding).

[0114] According to one embodiment, the processor (410) may receive a request from the user (510) to identify an external object corresponding to a target object. The processor (410) may identify an external object corresponding to the target object within the external environment (500). For example, the processor (410) may identify a visual object related to the external object corresponding to the target object within at least one image acquired during a specified time period. The processor (410) may highlight and display the visual object related to the external object. Accordingly, the processor (410) may identify an external object corresponding to the target object within the external environment and highlight and display a visual object related to the external object, thereby providing the user with information regarding the location of the external object corresponding to the target object. The operation of the wearable device (200) (or the processor (410)) according to the above-described embodiment will be described below.

[0115] FIG. 6 illustrates components of a wearable device for identifying an external object corresponding to a target object, according to one embodiment.

[0116] Referring to FIG. 6, the components illustrated in FIG. 6 represent units that process at least one function or operation, and may be implemented by hardware, software, or a combination of hardware and software. Depending on the embodiment, at least some of the components illustrated in FIG. 6 may be omitted. Depending on the embodiment, at least some of the components illustrated in FIG. 6 may be included in and operated by an external electronic device (e.g., a server, an electronic device for controlling a wearable device (200).

[0117] The processor (410) can identify visual information rendered in real time. The processor (410) can identify (or construct, acquire) a spatial map constructed within a specified time interval (or time-window). Based on the visual information or spatial map rendered in real time, a plurality of external objects within the external environment can be identified. The processor (410) can identify an external object corresponding to a target object among the plurality of external objects.

[0118] For example, the processor (410) may operate in a mode for identifying an external object corresponding to a target object among a plurality of objects within an external environment. This mode may be referred to as a visual search mode. The processor (410) may identify (or obtain) information about the target object within this mode. The processor (410) may identify an external object corresponding to the target object among a plurality of objects within the external environment using the components illustrated in FIG. 6.

[0119] The processor (410) may provide an image acquired through the camera (425) to another component using the camera data input unit (609). The image may be related to an external environment. The image may include a plurality of visual objects corresponding to a plurality of objects within the external environment.

[0120] The processor (410) can manage information about a target object using the query management unit (601). For example, the processor (410) can identify information about a target object based on a user input. The information about the target object can include information about the shape and / or visual pattern of the target object and information about the size of the target object. For example, the information about the shape and / or visual pattern of the target object can include an image containing the target object. The visual pattern can be referred to as a visual design, which can include labels, logos, text, images, and other features other than the shape. Depending on the embodiment, the information about the target object can further include depth information about the target object (or a visual object corresponding to the target object). An example of utilizing depth information about the target object (or a visual object corresponding to the target object) will be described below with reference to FIG. 8B. For example, the processor (410) can identify information about the depth of the target object using a ToF sensor. For example, the processor (410) may identify information regarding the depth of a target object based on images acquired based on the distance between two or more cameras spaced apart from each other. The processor (410) may identify information regarding the size of the target object based on the information regarding the depth of the target object.

[0121] The processor (410) may receive information about a target object from a user to search for an external object corresponding to the target object within an external environment. Information about the target object may be referred to as a query in the sense that it is used by the user to search for the target object. For example, information about the shape and / or visual pattern of the target object may be referred to as a main query. Information about the size of the target object may be referred to as an additional query. According to an embodiment, information for identifying an external object corresponding to the target object, excluding information about the shape and / or visual pattern of the target object, may be referred to as an additional query. For example, information about a designated time interval for identifying an external object corresponding to the target object may be referred to as an additional query.

[0122] According to an embodiment, the processor (410) may identify an external object corresponding to a target object using an external wearable device. The processor (410) may transmit information (or a query) regarding the target object to the external wearable device using a query transmission unit (602). According to an embodiment, the processor (410) may receive information (or a query) regarding the target object from the external wearable device using a query reception unit (603). A specific example of the processor (410) identifying an external object corresponding to the target object using an external wearable device will be specifically described in FIGS. 10 to 11B.

[0123] The processor (410) may identify candidate visual objects corresponding to the target object using the first model (604) within spatial information (e.g., a spatial map) constructed based on information about frames being rendered in real time within a specified time interval or information about simultaneous localization and mapping (SLAM). For example, the processor (410) may identify at least one feature point of the target object based on a main query (or information about the shape and / or visual pattern of the target object). The processor (410) may set the at least one feature point and spatial information of the target object as input values ​​of the first model (604). The processor (410) may identify candidate visual objects based on an output value of the first model (604). The processor (410) may also identify a visual object related to an external object corresponding to the target object based on an output value of the first model (604).

[0124] The processor (410) may store candidate visual objects using a buffer (605). The processor (410) may store the candidate visual objects to identify a visual object related to an external object corresponding to a target object among the candidate visual objects using a second model (606).

[0125] The processor (410) can identify a visual object related to an external object corresponding to a target object among candidate visual objects using the second model (606). The processor (410) can identify a visual object related to an external object corresponding to the target object among the candidate visual objects based on an additional query (e.g., information about the size of the target object). For example, the processor (410) can set the candidate visual objects and the additional query (e.g., information about the size of the target object) as input values ​​of the second model (606). The processor (410) can identify a visual object related to an external object corresponding to the target object based on an output value of the second model (606).

[0126] For example, each of the first model (604) and the second model (606) may be guided by multiple parameters. Each of the first model (604) and the second model (606) may be configured based on an artificial intelligence model. For example, the artificial intelligence model may correspond to the artificial intelligence model described in FIG. 1.

[0127] According to an embodiment, the processor (410) may transmit information about a visual object regarding an external object corresponding to a target object to an external wearable device using the result transceiver (608). The information about the visual object may include at least one of information about the location of the external object or information about the time at which the visual object was identified. The processor (410) may receive information about the visual object from the external wearable device using the result transceiver (608). For example, the information about the visual object may include information about a center point of the visual object, information about a bounding box of the visual object, or information about segmentation at the pixel level.

[0128] When an image including the visual object is displayed, the processor (410) may use the screen conversion unit (607) to change the image including the visual object to emphasize the visual object relative to an external object corresponding to the target object. When the image including the visual object is not displayed through the display (420), the processor (410) may use the screen conversion unit (607) to display an affordance for changing the user's gaze to display the image including the visual object.

[0129] The processor (410) can configure a screen to be displayed through the display (420) using the rendering unit (610). The processor (410) can configure a screen to be displayed through the display (420) using the rendering unit (610) based on an image (or affordance) acquired through the screen conversion unit (607) and data regarding the external environment acquired through the camera data input unit (609). The processor (410) can display the screen configured using the screen output unit (611) through the display (420).

[0130] Figure 7 illustrates a flowchart of the operation of a wearable device according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0131] Referring to FIG. 7, in operation 710, the processor (410) may identify information regarding a target object. For example, the processor (410) may set the mode of the wearable device (200) to a mode for identifying an external object corresponding to the target object. As an example, the processor (410) may set the mode of the wearable device (200) to a mode for identifying an external object corresponding to the target object based on a user input. The user input may include a specified motion input (e.g., eye movement, hand movement, movement of a controller connected to the wearable device (200)) and a specified voice input.

[0132] According to one embodiment, the processor (410) may identify information about the target object within a mode for identifying an external object corresponding to the target object. For example, the processor (410) may identify information about the target object based on a result searched through a network. The processor (410) may perform a search for the target object based on a user input. The processor (410) may identify information about the target object (e.g., information about the shape and / or visual pattern of the target object) based on identifying at least one target image of the target object. For example, the processor (410) may identify information about the target object based on information stored in the memory (415). As an example, the processor (410) may identify information about the target object based on an image of the target object identified through a gallery application. For example, the processor (410) may identify information about a target object based on identifying a file (e.g., text, image, video) related to the target object identified through a file browsing application.

[0133] For example, the processor (410) can identify additional information about the target object. Based on user input, the processor (410) can identify information about the size of the target object (e.g., 10 cm height). Based on user input, the processor (410) can identify information about a specified time period (e.g., the last 30 seconds) for identifying the target object.

[0134] In one embodiment, information about a target object can be used to find an external object corresponding to the target object. Information about the target object can be referenced through a query.

[0135] In operation 720, the processor (410) may identify a visual object relating to an external object corresponding to the target object within at least one image. For example, the processor (410) may identify a visual object relating to an external object corresponding to the target object within at least one image acquired based on the gaze of the user (510) of the wearable device (200) during a specified time period.

[0136] According to one embodiment, the processor (410) may identify (or acquire) at least one image based on the gaze of the user (510) of the wearable device (200) during a specified time period. For example, as the user's gaze changes, the image acquired through the camera (425) may change. The processor (410) may identify at least one image acquired as the user's gaze changes during the specified time period.

[0137] For example, the processor (410) may identify at least one image acquired prior to a specified time interval from the time at which information about the target object was identified. The processor (410) may identify a visual object related to an external object corresponding to the target object within the at least one image.

[0138] For example, at least one image may include images acquired based on a specified time interval during a specified time period. For example, at least one image may include images (e.g., frames) acquired consecutively during a specified time period. For example, at least one image may relate to an external environment. Each of the at least one image may represent a portion of the external environment.

[0139] According to one embodiment, the processor (410) may identify candidate visual objects corresponding to the target object based on information about the shape and / or visual pattern of the target object included in the information about the target object. The candidate visual objects may be configured based on the shape and / or visual pattern of the target object within at least one image. The processor (410) may identify a visual object related to an external object corresponding to the target object among the candidate visual objects based on information about the size of the target object included in the information about the target object. For example, each of the candidate visual objects may be a visual object cropped from at least one image.

[0140] According to one embodiment, the processor (410) can obtain information about a plurality of visual objects included in at least one image. The processor (410) can store the information about the plurality of visual objects in the memory (415). The processor (410) can set the information about the plurality of visual objects and the information about the target object as input values ​​of a designated model (e.g., the first model (604) or the second model (606) of FIG. 6) indicated by a plurality of parameters. The processor (410) can identify a visual object related to an external object corresponding to the target object within at least one image based on an output value of the designated model.

[0141] According to one embodiment, the processor (410) may identify spatial information about an external object based on at least one image. For example, the processor (410) may identify a spatial map based on at least one image. The processor (410) may identify a visual object about an external object corresponding to a target object based on the identified spatial map. The processor (410) may identify a location of a visual object about an external object corresponding to the target object based on the identified spatial map.

[0142] According to one embodiment, the processor (410) can identify a plurality of external objects of various classes (or undetermined classes) on a spatial map based on at least one image identified during a specified time interval. The processor (410) can identify candidate visual objects corresponding to a target object using a specified model (e.g., the first model (604) of FIG. 6). The processor (410) can identify external objects corresponding to the candidate visual objects within the spatial map. For example, the processor (410) can identify candidate visual objects (or visual objects related to the external object) within the at least one image based on image feature similarity.

[0143] For example, the processor (410) can identify feature points of each of a plurality of external objects (or a plurality of visual objects related to the plurality of external objects). The processor (410) can identify feature points of a target object (or an image related to the target object). The processor (410) can identify candidate visual objects based on identifying the similarity between the feature points of each of the plurality of external objects and the feature points of the target object.

[0144] As described above, the processor (410) can identify candidate visual objects based on identifying similarities between two or more multi-dimensional vectors (e.g., feature points of each of a plurality of external objects and feature points of a target object).

[0145] According to an embodiment, the processor (410) may identify images of the target object viewed from various directions based on an image of the target object. The processor (410) may identify a visual object similar to one of the images of the target object viewed from various directions as one of the candidate visual objects.

[0146] The processor (410) can store each of the candidate visual objects in the form of coordinates within the corresponding image (or spatial map) in a memory (415) (e.g., buffer (605) of FIG. 6). The processor (410) can identify a visual object related to an external object corresponding to a target object among the candidate visual objects using a designated model (e.g., second model (606) of FIG. 6).

[0147] In operation 730, the processor (410) may identify whether a first image including a visual object related to an external object is displayed through the display (420). For example, the processor (410) may identify whether the first image including the visual object is displayed through the display (420) based on the gaze of the user (510).

[0148] The processor (410) can identify a visual object related to an external object corresponding to a target object within at least one image acquired during a specified time period. Since at least one image has been identified in the past, the visual object may not be included in the image currently displayed through the display (420). Accordingly, the processor (410) can identify whether a first image including a visual object is displayed through the display (420) based on the gaze of the user (510).

[0149] For example, the first image may refer to an image including a visual object. The fact that the first image is displayed through the display (420) may mean that an external object corresponding to the target object exists within the field of view (FoV) of the camera (425) of the wearable device (200).

[0150] In operation 740, when a first image including a visual object related to an external object corresponding to a target object is displayed through the display (420), the processor (410) may change the first image to emphasize the visual object related to the external object. For example, the processor (410) may change the first image to emphasize the visual object related to the external object based on identifying that the first image including the visual object is displayed through the display (420).

[0151] For example, the processor (410) may change the display of an area in the first image except for an area corresponding to the visual object based on identifying that a first image including a visual object related to an external object is displayed through the display (420). For example, the processor (410) may maintain the color of the area corresponding to the visual object in the first image and set the color of the remaining area to black and white or grayscale. For example, the processor (410) may omit an area in the first image except for an area corresponding to the visual object. For example, the processor (410) may identify data for an area corresponding to a visual object in the first image and, based on the identified data, display only the visual object. For example, the processor (410) may display the first image by emphasizing the borders of the visual object.

[0152] For example, the processor (410) may display an element for highlighting a visual object related to an external object within the first image. As an example, the processor (410) may display a bounding box for a visual object related to the external object within the first image. As an example, the processor (410) may display an element (e.g., an arrow, a circle, a polygon, an exclamation mark) indicating a visual object related to the external object.

[0153] According to one embodiment, the processor (410) can identify a distance between an external object and a wearable device (200). The processor (410) can display information about the distance between the external object and the wearable device (200) in conjunction with a visual object related to the external object. For example, the processor (410) can display text indicating the distance between the external object and the wearable device (200) together with a visual object related to the external object.

[0154] In operation 750, if a first image including a visual object relating to an external object corresponding to a target object is not displayed through the display (420), the processor (410) may display an affordance for changing the gaze of the user (510) to display the first image including a visual object relating to the external object, superimposed on a second image. For example, the processor (410) may display an affordance for changing the gaze of the user (510) to display the first image, superimposed on a second image, based on identifying that a second image distinct from the first image including a visual object relating to the external object is displayed through the display (420). For example, the second image may be referenced as an image that does not include a visual object relating to the external object. For example, the affordance for changing the gaze of the user (510) may include an arrow, a mini-map, or a pop-up window.

[0155] According to one embodiment, the processor (410) may display a first image including a visual object related to an external object through the display (420) based on a change in the gaze of the user (510) toward an external object corresponding to a target object. The processor (410) may perform operation 740 based on a change in the second image to the first image.

[0156] FIG. 8A illustrates an example of operation of a wearable device according to one embodiment.

[0157] Referring to FIG. 8A, the wearable device (200) may be positioned to face the external environment (800). The user (510) of the wearable device (200) may be in a state of looking at the external environment (800). The processor (410) may obtain data regarding the external environment (800) using the camera (425). Based on the data regarding the external environment (800), the processor (410) may obtain an image (810) regarding the external environment (800). The processor (410) may display the image (810) through the display (420).

[0158] For example, the external environment (800) may include multiple external objects. For example, the external environment (800) may be a market including multiple products. The image (810) may include multiple visual objects corresponding to the multiple products. The user (510) may set the mode of the wearable device (200) to a mode for identifying an external object corresponding to a target object.

[0159] According to one embodiment, the processor (410) may set the mode of the wearable device (200) to a mode for identifying an external object corresponding to a target object (e.g., a target product). Within the mode for identifying an external object (e.g., an actual product) corresponding to the target object, the processor (410) may identify information about the target object. For example, the processor (410) may identify information about the target object based on a user input.

[0160] For example, the processor (410) may identify a user input for executing a gallery application. The processor (410) may display an image (820) that is superimposed on the image (810) in a user interface (821) for the gallery application. The processor (410) may identify an input for selecting an image (822) for a target object (e.g., C cereal) within the user interface (821) for the gallery application. Based on the identified input, the processor (410) may identify information about the target object. Based on the image (822) for the target object, the processor (410) may identify information about the shape and / or visual appearance of the target object. Furthermore, the processor (410) may perform visual pattern matching that adjusts for changes in direction, lighting, and scale to identify the most likely match.

[0161] The processor (410) can identify feature points of an image (822) regarding a target object. The processor (410) can identify feature points of each of a plurality of visual objects included in the image (810). The processor (410) can set the feature points of each of the plurality of visual objects and the feature points of the image (822) as input values ​​of a designated model (e.g., the first model (604) or the second model (606) of FIG. 6). Based on the output value of the designated model, the processor (410) can identify a visual object (831) for an external object corresponding to the target object.

[0162] The processor (410) can display an image (830) through the display (420) based on identifying a visual object (831). The processor (410) can display an image (830) in which the visual object (831) is highlighted.

[0163] For example, the processor (410) can remove the color of the remaining area except for the area corresponding to the visual object (831). The processor (410) can maintain the color of the area corresponding to the visual object (831) and display the color of the remaining area based on a gray scale.

[0164] For example, the processor (410) may display a bounding box (832) in an area corresponding to a visual object (831). The processor (410) may display the bounding box (832) overlapping the visual object (831). The positioning of the bounding box (832) may be adjusted as the user (510) moves relative to the visual object (831).

[0165] For example, the processor (410) can change the color of the visual object (831) (or an area corresponding to the visual object (831)). The processor (410) can identify the confidence level regarding the output value of the specified model. The processor (410) can change the color of the visual object (831) based on the confidence level. For example, when the confidence level is low, the processor (410) can change the color of the visual object (831) to a color similar to the surrounding color(s) of the visual object (831). When the confidence level is high, the processor (410) can change the color of the visual object (831) to a color that is distinct from the surrounding color(s) of the visual object (831).

[0166] According to an embodiment, the processor (410) may identify a distance between an external object and a wearable device (200). The processor (410) may display information regarding the distance between the external object and the wearable device (200) through a display (420) together with a visual object (831). The image (830) may include text (833) indicating the distance between the external object and the wearable device (200). The text (833) may be displayed in conjunction with the visual object (831).

[0167] In one embodiment, a user (510) may identify a visual object (831) regarding an external object corresponding to a target object. The processor (410) may identify a user input for terminating a mode for identifying an external object corresponding to the target object while the image (830) is displayed through the display (420). Based on the user input, the processor (410) may change the image displayed through the display (420) from the image (830) to the image (840). The image (840) may correspond to the image (810).

[0168] FIG. 8b illustrates an example of operation of a wearable device according to one embodiment.

[0169] Referring to FIG. 8B, the wearable device (200) may be positioned to face a first portion (851) of the external environment (850) based on a direction (d1). A user (510) of the wearable device (200) may be in a state of looking at the first portion (851) of the external environment (850). The processor (410) may obtain data regarding the first portion (851) of the external environment (850) using a camera (425). The processor (410) may obtain an image (860) regarding the first portion (851) of the external environment (850) based on the data regarding the first portion (851) of the external environment (850). The processor (410) may display the image (860) through the display (420).

[0170] For example, the processor (410) may have acquired an image regarding a second portion (852) of the external environment (850). The processor (410) may store information regarding a plurality of external objects included in the second portion (852) of the external environment (850) in the memory (415). For example, the processor (410) may store information regarding an external object (854) in the memory (415).

[0171] According to one embodiment, the user (510) may set the mode of the wearable device (200) to a mode for identifying an external object corresponding to a target object. For example, the processor (410) may set an external object (853) as a target object while an image (860) is displayed through the display (420). The processor (410) may set the external object (853) as the target object based on an input for a visual object (861) corresponding to the external object (853).

[0172] Within the mode for identifying an external object corresponding to a target object, the processor (410) may identify information about the target object. For example, the processor (410) may identify information about the target object based on a visual object (861). The processor (410) may identify information about the shape and / or visual pattern of the target object based on the visual object (861).

[0173] For example, the processor (410) can identify information about the depth of the visual object (861). Based on the information about the depth of the visual object (861), the processor (410) can identify information about the size of the external object (854) (or the target object). As an example, the processor (410) can identify information about the depth of the visual object (861) using a ToF sensor. As an example, the processor (410) can identify information about the depth of the visual object (861) based on images acquired based on the distance between two or more cameras spaced apart from each other. Based on the information about the depth of the visual object (861), the processor (410) can identify information about the size of the external object (854) (or the target object).

[0174] For example, the processor (410) can identify images of the target object viewed from various directions based on the visual object (861). The processor (410) can identify an external object (854) based on images of the target object viewed from various directions.

[0175] For example, the processor (410) may identify an external object (854) corresponding to an external object (853) that is a target object within the external environment (850). The external object (854) may be included in a second portion (852) of the external environment (850). The processor (410) may display an image (870) through the display (420) that includes an affordance (871) (e.g., a suggestion or prompt in graphical and / or textual form) for changing the direction of the user's (510) gaze to face the direction (d2). Although the affordance (871) within the image (870) is configured as an arrow, this is exemplary and not limited thereto. Although not illustrated, the affordance (871) may be configured as a mini-map, an exclamation mark, or a color.

[0176] After an image (870) including an affordance (871) is displayed through the display (420), the processor (410) can identify that the direction toward which the wearable device (200) (or the gaze of the user (510)) is directed has changed from direction (d1) to direction (d2).

[0177] The processor (410) may display an image (880) regarding a second part (852) of the external environment (850) based on the direction in which the wearable device (200) is facing being the direction (d2). The processor (410) may display an image (880) including a visual object (881) regarding an external object (854) corresponding to a target object (i.e., the external object (853)) through the display (420). For example, the processor (410) may change the image (880) to emphasize the visual object (881). The processor (410) may change the image (880) based on maintaining the color of an area corresponding to the visual object (881) within the image (880) and removing the color of the remaining areas.

[0178] FIG. 9 illustrates an example of operation of a wearable device according to one embodiment.

[0179] Referring to FIG. 9, the wearable device (200) may be positioned to face the external environment (900). The user (510) of the wearable device (200) may be in a state of looking at the external environment (900). The processor (410) may obtain data regarding the external environment (900) using the camera (425). Based on the data regarding the external environment (900), the processor (410) may obtain an image (910) regarding the external environment (900). The processor (410) may display the image (910) through the display (420).

[0180] The processor (410) may set the mode of the wearable device (200) to a mode for identifying an external object corresponding to a target object. Within the mode for identifying an external object corresponding to a target object, the processor (410) may identify information about the target object. For example, the processor (410) may identify information about the target object based on a user input.

[0181] For example, the processor (410) may identify a user input for executing a gallery application. Based on the user input, the processor (410) may cause a user interface (921) for the gallery application to display an image (920) that is superimposed on the image (910). The processor (410) may identify an input for selecting an image (922) for a target object (e.g., a wine glass) within the user interface (921) for the gallery application. Based on the identified input, the processor (410) may identify information about the target object. Based on the image (922) for the target object, the processor (410) may identify information about the shape and / or visual pattern of the target object.

[0182] The processor (410) can identify feature points of an image (922) (or target object) regarding a target object. The processor (410) can identify feature points of each of a plurality of visual objects included in the image (910). The processor (410) can set the feature points of each of the plurality of visual objects and the feature points of the image (922) as input values ​​of a designated model (e.g., the first model (604) or the second model (606) of FIG. 6). Based on the output values ​​of the designated model, the processor (410) can identify a visual object (931) and a visual object (932) for an external object corresponding to the target object.

[0183] For example, each of the visual objects (931) and (932) may be referred to as candidate visual objects. The processor (410) may display an image (930) including the visual objects (931) and (932) that are candidate visual objects. The processor (410) may display an element (e.g., a bounding box) on the image (930) that indicates that the visual objects (931) and (932) are candidate visual objects.

[0184] The processor (410) can identify information about the size of a target object while an image (930) including a visual object (931) and a visual object (932), which are candidate visual objects, is displayed through the display (420). For example, the processor (410) can identify information about the size of the target object based on a user input.

[0185] For example, the processor (410) can identify information about the size of a target object based on a voice input such as "Find a wine glass with a height of 20 cm." The external object (901) can be a picture of a wine glass in a poster. The external object (902) can be an actual wine glass. The height of the external object (901) can be about 10 cm. The height of the external object (902) can be about 20 cm. The processor (410) can identify an external object (902) corresponding to the target object based on the information about the size of the target object. The processor (410) can display an image (940) that emphasizes a visual object (932) for the external object (902).

[0186] In some embodiments, the processor (410) may identify information about the size of the target object together with an input for selecting an image (922) of the target object (e.g., a wine glass).

[0187] According to one embodiment, the processor (410) may identify candidate visual objects (e.g., visual object (931), visual object (932)) that are similar to the shape and / or visual pattern of the target object based on a first model included in the specified model. The processor (410) may identify external objects (e.g., external object (901), external object (902)) that are similar to the size of the target object among external objects corresponding to the candidate visual objects (e.g., visual object (931), visual object (932)) based on a second model included in the specified model.

[0188] As described above, the processor (410) can identify candidate visual objects based on a visual query (or information about the shape and / or visual pattern of the target object), and additionally, identify a visual object related to an external object corresponding to the target object among the candidate visual objects based on the identified scale information (or information about the size of the target object). The processor (410) can refine the identified result based on the visual query and provide it to the user. For example, the processor (410) can identify (or estimate) local depth or 3D (3-dimensional) scale information for the candidate visual object using stereo matching or a time of flight (ToF) sensor, without estimating depth for at least one entire image. The processor (410) can reduce the amount of computation based on identifying local depth or 3D (3-dimensional) scale information for the candidate visual object.

[0189] FIG. 10 illustrates an example of operation of a wearable device according to one embodiment.

[0190] Referring to FIG. 10, the camera (425) of the wearable device (200) may be configured with one or more cameras. For example, the wearable device (200) may include a first camera for composing an image (or screen) to be displayed through the display (420), and a second camera and a third camera for identifying the surroundings of the user (510) but not displayed through the display (420). For example, the second camera and the third camera may be referred to as supplementary side-facing cameras. For convenience of explanation, an example in which the camera (425) includes the first camera, the third camera, and the like is described below, but is not limited thereto. For example, the camera (425) may be configured as a wide-angle camera.

[0191] For example, the processor (410) may obtain data about a first part (1001) of the external environment (1000) using a first camera. Based on the data about the first part (1001) of the external environment (1000), the processor (410) may display an image (not shown) corresponding to the first part (1001) of the external environment (1000) through the display (420). For example, while the processor (410) obtains data about the first part (1001) of the external environment (1000) using the first camera, the processor (410) may obtain data about the second part (1002) of the external environment (1000) using a second camera, and may obtain data about the third part (1003) of the external environment (1000) using a third camera. The processor (410) can obtain an image corresponding to the second part (1002) of the external environment (1000) based on data about the second part (1002) of the external environment (1000). The processor (410) can obtain an image corresponding to the third part (1003) of the external environment (1000) based on data about the third part (1003) of the external environment (1000).

[0192] While the processor (410) displays an image (not shown) corresponding to a first part (1001) of the external environment (1000) through the display (420), the processor (410) may store an image corresponding to a second part (1002) of the external environment (1000) and an image corresponding to a third part (1003) of the external environment (1000) in the memory (415). For example, since the image corresponding to the second part (1002) of the external environment (1000) and the image corresponding to the third part (1003) of the external environment (1000) are not displayed through the display (420), rendering may not be performed for the image corresponding to the second part (1002) of the external environment (1000) and the image corresponding to the third part (1003) of the external environment (1000).

[0193] According to one embodiment, the processor (410) may set the mode of the wearable device (200) to a mode for identifying an external object corresponding to a target object. Within the mode for identifying an external object corresponding to the target object, the processor (410) may identify information about the target object. For example, the processor (410) may identify information about the target object based on a user input.

[0194] For example, the processor (410) may perform a search for a target object through a network based on a user input (e.g., voice input, keyboard input). The processor (410) may display an image (1010) in which a user interface (1011) including a plurality of images according to the search results is superimposed on an image (not shown) corresponding to a first portion (1001) of the external environment (1000).

[0195] The processor (410) can identify a user input for selecting at least one target image of a target object from among a plurality of images displayed within the user interface (1011). The processor (410) can identify at least one target image of a target object from among a plurality of images displayed within the user interface (1011). For example, the at least one target image can include image (1012) and image (1013).

[0196] The processor (410) can identify information about a target object based on at least one target image (e.g., image (1012) and image (1013)). The processor (410) can identify information about a shape and / or visual pattern of the target object based on at least one target image.

[0197] According to an embodiment, when an input for selecting a target image is received from a user (510), the processor (410) may perform an image search based on the target image. The processor (410) may acquire images of the target object viewed from various directions through a network. The processor (410) may identify a specified number of images among the images viewed from various directions based on similarity. The processor (410) may set a weight for each of the specified number of images based on the similarity. The processor (410) may identify a visual object for an external object corresponding to the target object based on the weight. According to the embodiment, the processor (410) may identify a visual object representing the back of the specified car among the images acquired through the camera (425) based on identifying an image of the front of the specified car.

[0198] According to an embodiment, the processor (410) may identify a visual object regarding an external object corresponding to a target object within images acquired through the camera (425) based on at least one target image. For example, the processor (410) may identify a visual object regarding an external object corresponding to a target object within an image corresponding to a first portion (1001) of the external environment (1000), an image corresponding to a second portion (1002) of the external environment (1000), and an image corresponding to a third portion (1003) of the external environment (1000).

[0199] The processor (410) can identify that the external object (1050) corresponding to the target object is included in the third portion (1003) of the external environment (1000). For example, the processor (410) can identify that the external object (1050) corresponding to the target object is included in the third portion (1003) of the external environment (1000) based on identifying a visual object (e.g., a visual object (1031) in the image (1030)) regarding the external object (1050) corresponding to the target object in an image corresponding to the third portion (1003) of the external environment (1000). The processor (410) can display an image (1020) corresponding to a first part (1001) of the external environment (1000) and an affordance (1021) for guiding the user (510) to change his / her gaze toward a third part (1003) of the external environment (1000) through the display (420).

[0200] The processor (410) may display an image (1030) including a visual object (1031) regarding an external object (1050) corresponding to a target object through the display (420) based on identifying that the user's (510) gaze has changed toward a third portion (1003) of the external environment (1000). The visual object (1031) may be emphasized within the image (1030). For example, the processor (410) may remove the color of an area other than an area corresponding to the visual object (1031). The processor (410) may maintain the color of the area corresponding to the visual object (1031) and display the color of the remaining area based on a gray scale.

[0201] The processor (410) can identify a user input to terminate a mode for identifying an external object corresponding to a target object while the image (1030) is displayed through the display (420). For example, the user input can include a specified motion input (e.g., eye movement, hand movement, movement of a controller connected to the wearable device (200)) and a specified voice input.

[0202] The processor (410) can change the image displayed through the display (420) from image (1030) to image (1040) based on the user input. The processor (410) can change the image displayed through the display (420) from image (1030) to image (1040) by stopping the operation of highlighting the visual object (1031).

[0203] According to one embodiment, the processor (410) may identify additional images based on the movement of the user (510) while the wearable device (200) is operating in a mode for identifying an external object corresponding to a target object, regardless of whether an external object corresponding to the target object is identified. The processor (410) may highlight a visual object within the additional images based on whether a visual object related to the external object corresponding to the target object is identified.

[0204] According to one embodiment, a first area of ​​the external environment identified through the camera (425) may be set larger than a second area of ​​the external environment displayed through the display (420). For example, the camera (425) may identify multiple surrounding areas of the wearable device (200). Even when a visual object corresponding to the external object corresponding to the target object is not displayed in an image displayed through the display (420), the processor (410) may identify an external object corresponding to the target object within the surrounding area of ​​the wearable device (200). The processor (410) may provide an affordance as a prompt or instructions to direct the user's gaze toward the external object when the user's gaze follows the affordance. According to an embodiment, the processor (410) may obtain information (e.g., a spatial map) about the external environment in which the wearable device (200) is located from an external electronic device (e.g., an external wearable device, a server). Based on the information about the external environment, the processor (410) may identify an external object corresponding to a target object within the external environment.

[0205] Figure 11 illustrates a flowchart of the operation of a wearable device according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0206] Referring to FIG. 11, the wearable device (200) can identify another wearable device (hereinafter, external wearable device) connected to the wearable device (200) (e.g., connected via communication). According to one embodiment, the wearable device (200) can identify an external wearable device located within a specified distance from the wearable device (200). According to one embodiment, the wearable device (200) can identify an external wearable device located in the same space (e.g., home, market) as the wearable device (200). According to one embodiment, the wearable device (200) can identify an external wearable device located within the FoV of the camera (425) of the wearable device (200). For example, the processor (410) can establish a connection with the external wearable device. The processor (410) can perform operations 1110 to 1140 while a connection with an external wearable device is established.

[0207] In operation 1110, the processor (410) may identify information about a target object. For example, the processor (410) may identify information about the target object within a mode for identifying an external object corresponding to the target object. Operation 1110 may correspond to operation 710 of FIG. 7 .

[0208] In operation 1120, the processor (410) may request the external wearable device to identify a visual object related to an external object corresponding to the target object. For example, the processor (410) may request the external wearable device to identify a visual object related to the external object based on transmitting information about the target object to the external wearable device. For example, the information about the target object may include information about the shape and / or visual pattern of the target object and / or information about the size of the target object. For example, the processor (410) may transmit a query about information about the target object to the external wearable device.

[0209] The external wearable device can identify an external object relative to the target object based on information about the target object received from the wearable device (200). For example, the external wearable device can identify a visual object relative to the external object corresponding to the target object within at least one image acquired from the external wearable device based on operation 720 illustrated in FIG. 7.

[0210] For example, an external wearable device can accumulate SLAM information and store it in its memory. The external wearable device can accumulate SLAM information that changes over time and store it in its memory. Based on the SLAM information stored in its memory, the external wearable device can identify a visual object corresponding to an external object corresponding to a target object. Based on the SLAM information, the external wearable device can also identify the time at which the visual object was identified (e.g., 3 seconds ago, now, or at a specific time).

[0211] In operation 1130, the processor (410) may receive information regarding a visual object identified through an external wearable device. For example, the information regarding the visual object may include at least one of information regarding the location of the external object and information regarding the time at which the visual object was identified. The information regarding the location of the external object may include information regarding the three-dimensional spatial location of the external object. For example, the information regarding the time at which the visual object was identified may include information regarding the time (or timing) at which the visual object was identified within information regarding SLAM identified during a specified time interval according to a change (or movement) in the gaze of the user of the external wearable device. Based on the information regarding the time at which the visual object was identified, the processor (410) may identify the time at which the external object was captured through a camera of the external wearable device, in the external wearable device.

[0212] An external wearable device can identify a three-dimensional spatial location of an external object on a spatial map identified by the external wearable device. The processor (410) can transmit information about the three-dimensional spatial location of the external object to the wearable device (200). For example, the information about the three-dimensional spatial location can be configured as coordinates set based on the external wearable device. The external wearable device can transmit information about the positional relationship between the external wearable device and the wearable device (200) and information about the three-dimensional spatial location to the wearable device (200).

[0213] For example, information about a three-dimensional spatial location may be composed of fixed coordinates (e.g., world coordinates) set within a space including the wearable device (200) and an external wearable device.

[0214] According to one embodiment, the processor (410) may identify information about an external space including a first location where the wearable device (200) is located and a second location where an external wearable device is located based on information about a visual object. For example, the processor (410) may identify a spatial map of the external space including the first location where the wearable device (200) is located and the second location where the external wearable device is located based on information about a visual object. The processor (410) may identify a location of an external object within the spatial map of the external space. The processor (410) may identify a location of an external object as a third location within the spatial map of the external space.

[0215] In operation 1140, the processor (410) may display an element for indicating a location of an external object. For example, the processor (410) may identify a path from a first location where the wearable device (200) is located to a third location identified as the location of the external object based on information about the location of the external object. The processor (410) may display an element for indicating the location of the external object based on the identified path. For example, the processor (410) may display a spatial map of an external space through the display (420). The processor (410) may display a path from the first location to the third location on the spatial map. For example, the processor (410) may display a two-dimensional map based on the spatial map. The processor (410) may display a path from the first location to the third location on the two-dimensional map. For example, the processor (410) may display a direction and distance from the first location to the third location.

[0216] In operations 1110 through 1140, the operation of receiving information about a visual object from a single external wearable device is described, but is not limited thereto. The processor (410) can transmit information about a target object to multiple external electronic devices. The processor (410) can receive information about a visual object from each of the multiple external electronic devices.

[0217] FIG. 12A illustrates an example of an operation of a wearable device to identify an external object corresponding to a target object through an external wearable device, according to one embodiment.

[0218] FIG. 12b illustrates an example of an operation of a wearable device to identify an external object corresponding to a target object through an external wearable device, according to one embodiment.

[0219] Referring to FIG. 12A, the processor (410) can identify information about a target object (1201). For example, the processor (410) can obtain information (e.g., a target image) about the target object (1201) based on a search for the target object (1201) through a network. For example, the processor (410) can obtain information about the target object (1201) from an external electronic device. For example, the processor (410) can obtain information about the target object (1201) based on a user input for selecting at least one of a plurality of images displayed in a gallery application. The processor (410) can identify information about the target object (1201) stored in the memory (415) (or the cloud).

[0220] The processor (410) can transmit information about the target object (1201) to the external wearable device (1250). Based on transmitting the information about the target object (1201) to the external wearable device (1250), the processor (410) can request identification (or search) of a visual object about an external object corresponding to the target object (1201). The processor (410) can request identification of whether a visual object about an external object corresponding to the target object (1201) is included in at least one image identified by the external wearable device (1250) during a specified period of time.

[0221] The external wearable device (1250) can be used by a user (1251) distinct from the user (510) of the wearable device (200). The external wearable device (1250) can identify an external object relative to the target object (1201) based on information about the target object (1201). The external wearable device (1250) can identify a visual object relative to the external object corresponding to the target object (1201) within at least one image acquired through a camera of the external wearable device (1250).

[0222] For example, the external wearable device (1250) can identify a visual object (1211) regarding an external object corresponding to a target object (1201) within an image (1210). The external wearable device (1250) can identify a location of the external object. The external wearable device (1250) can identify the location of the external object as a location (1212).

[0223] The external wearable device (1250) can transmit information about a visual object (1211) to the wearable device (200). For example, the information about the visual object can include at least one of information about the location of the external object or information about the time at which the visual object (1211) was identified. For example, the information about the location of the external object can include information about the three-dimensional spatial location of the external object. The external wearable device (1250) can identify information about the three-dimensional spatial location of the external object on a spatial map of the external space (1200). The external wearable device (1250) can transmit information about the three-dimensional spatial location of the external object to the wearable device (200). For example, the external wearable device (1250) can identify the location of the external object as the location (1212). An external wearable device (1250) can transmit information about a location (1212) to the wearable device (200).

[0224] The processor (410) can receive information about a visual object (1211) from an external wearable device (1250). Based on the information about the visual object (1211), the processor (410) can identify the location of the external object as a location (1212). Based on the information about the visual object (1211), the processor (410) can identify the time at which the external object was identified.

[0225] According to one embodiment, the processor (410) may display at least one of an image (1221) or an image (1222) through the display (420) based on information about the visual object (1211).

[0226] For example, the processor (410) can identify information about an external space (1200) including a location (1231) where the wearable device (200) is located and a location (1253) where an external wearable device (1250) is located based on information about a visual object (1211). The processor (410) can identify a path from the location (1231) of the wearable device (200) to the location (1212) of the external object. The processor (410) can display an image (1221) representing a path from the location (1231) of the wearable device (200) to the location (1212) of the external object through the display (420) within a two-dimensional map corresponding to the external space (1200).

[0227] For example, the processor (410) can identify information about the distance and direction from the location (1231) of the wearable device (200) to the location (1212) of the external object based on information about the visual object (1211). The processor (410) can display an image (1222) representing information about the distance and direction from the location (1231) of the wearable device (200) to the location (1212) of the external object through the display (420).

[0228] Although not shown, in one embodiment, the processor (410) may simultaneously display an image (1221) and an image (1222) through the display (420). In one embodiment, the processor (410) may also display information about an external wearable device (1250) that has identified an external object and information about the location of the external wearable device (1250) through the display (420).

[0229] Referring to FIG. 12b, the processor (410) can identify information about the target object (1201). For example, the processor (410) can obtain information about the target object (1201) (e.g., a target image) based on a search for the target object (1201) through a network.

[0230] The processor (410) can transmit information about the target object (1201) to the external wearable device (1250). Based on transmitting the information about the target object (1201) to the external wearable device (1250), the processor (410) can request identification (or search) of a visual object about an external object corresponding to the target object (1201). For example, the processor (410) can identify (or set) a time interval for identifying a visual object about an external object corresponding to the target object (1201). The processor (410) can transmit information about the identified time interval to the external wearable device (1250).

[0231] The external wearable device (1250) can identify an external object relative to the target object (1201) based on information about the target object (1201). For example, the external wearable device (1250) can identify at least one image acquired using a camera during a specified time period. The external wearable device (1250) can identify (or acquire, generate) a spatial map based on at least one image acquired during the specified time period.

[0232] For example, while the external wearable device (1250) is located at a location (1253), the external wearable device (1250) can identify a visual object (1211) based on an image acquired at a first point in time within a specified time interval. The external wearable device (1250) can identify that an external object corresponding to the visual object (1211) at the first point in time is located at a location (1212) within the external space (1200). While the external wearable device (1250) is located at a location (1253), the external wearable device (1250) can identify a visual object (1211) based on an image (1260) acquired at a second point in time within a specified time interval. The external wearable device (1250) can identify that an external object corresponding to a visual object (1211) at a second point in time is located at a location (1213) within the external space (1200). Based on identifying that the visual object (1211) has moved, the external wearable device (1250) can identify that the location of the external object corresponding to the visual object (1211) has changed from location (1212) to location (1213).

[0233] While the external wearable device (1250) is located at the location (1254), the external wearable device (1250) can identify a visual object (1215) based on an image acquired at a third point in time within a designated time interval. The external wearable device (1250) can identify that another external object corresponding to the visual object (1215) is located at a location (1214) within the external space (1200) at the third point in time. For example, the shape or visual pattern of the other external object corresponding to the visual object (1215) may be the same as or similar to the shape or visual pattern of the external object corresponding to the visual object (1211). The size of the other external object corresponding to the visual object (1215) may be different from the size of the external object corresponding to the visual object (1211). For example, the other external object may be a picture of the external object. For example, the other external object may be a miniature of the external object.

[0234] The external wearable device (1250) can identify a spatial map based on at least one image acquired during a specified time interval. For example, the external wearable device (1250) can identify a spatial map based on information indicating that an external object is located at a location (1213) at a second time point and information indicating that another external object is located at a location (1214) at a third time point.

[0235] The external wearable device (1250) can identify a visual object (1211) regarding an external object corresponding to the target object (1201) based on receiving information regarding the target object (1201) from the wearable device (200). For example, the external wearable device (1250) can identify the visual object (1211) and the visual object (1215) as candidate visual objects based on information regarding the shape and / or visual pattern of the target object (1201). The external wearable device (1250) can identify the visual object (1211) regarding the external object corresponding to the target object (1201) among the candidate visual objects based on information regarding the size of the target object (1201).

[0236] The processor (410) can receive information about a visual object (1211) from an external wearable device (1250). Based on the information about the visual object (1211), the processor (410) can identify the location of the external object as a location (1213). Based on the information about the visual object (1211), the processor (410) can identify a second point in time at which the external object is identified.

[0237] According to one embodiment, the processor (410) may display at least one of the image (1261) or the image (1262) through the display (420) based on information about the visual object (1211).

[0238] For example, the processor (410) can identify information about an external space (1200) including a location (1231) where the wearable device (200) is located and a location (1254) where an external wearable device is located based on information about a visual object (1211). The processor (410) can identify a path from the location (1231) of the wearable device (200) to the location (1213) of the external object. The processor (410) can display an image (1261) representing a path from the location (1231) of the wearable device (200) to the location (1213) of the external object through the display (420) within a two-dimensional map corresponding to the external space (1200).

[0239] For example, the processor (410) can identify information about the distance and direction from the location (1231) of the wearable device (200) to the location (1213) of the external object based on information about the visual object (1211). The processor (410) can display an image (1262) representing information about the distance and direction from the location (1231) of the wearable device (200) to the location (1213) of the external object through the display (420).

[0240] Although not shown, in one embodiment, the processor (410) may simultaneously display the image (1261) and the image (1262) through the display (420). In one embodiment, the processor (410) may also display information about an external wearable device (1250) that has identified an external object and information about the location of the external wearable device (1250) through the display (420).

[0241] According to one embodiment, a wearable device (e.g., wearable device (200)) may include a display (e.g., display (420)), one or more cameras (e.g., camera (425)), a memory (e.g., memory (415)) including one or more storage media for storing instructions, and at least one processor (e.g., processor (410)) including processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify information about a target object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a visual object, relative to an external object corresponding to the target object, within at least one image acquired by the one or more cameras and acquired based on a gaze of a user of the wearable device during a designated time period. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, based on the gaze of the user, whether a first image including the visual object is displayed via the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to alter the first image to emphasize the visual object based on identifying that the first image including the visual object is displayed via the display.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display an affordance for shifting a user's gaze to display the first image superimposed on the second image based on identifying that a second image distinct from the first image including the visual object is displayed through the display.

[0242] According to one embodiment, the information about the target object may include information about a shape and / or a visual pattern of the target object and information about a size of the target object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify one or more candidate visual objects corresponding to the target object based on the information about the shape and / or visual pattern of the target object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify the visual object among the one or more candidate visual objects based on the information about the size of the target object.

[0243] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify at least one target image of the target object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify information about a shape and / or a visual pattern of the target object based on the at least one target image.

[0244] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify information regarding the size of the target object based on a user input.

[0245] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to change the display of an area within the first image other than an area corresponding to the visual object, based on identifying that the first image including the visual object is displayed through the display.

[0246] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display information regarding a distance between the external object and the wearable device in association with the visual object.

[0247] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to store information about a plurality of visual objects included in the at least one image in the memory based on obtaining information about the plurality of visual objects. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to set information about the plurality of visual objects and information about the target object as input values ​​of a designated model indicated by a plurality of parameters. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify the visual object in relation to the external object corresponding to the target object within the at least one image based on an output value of the designated model.

[0248] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to acquire an image of an external environment including the first image and the second image using the one or more cameras. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display, one of the first image and the second image based on a portion of the image of the external environment.

[0249] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to request an external wearable device connected to the wearable device to identify the visual object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive information about the visual object identified through the external wearable device based on the request. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display an element for indicating a location of the external object based on the information about the visual object.

[0250] According to one embodiment, the information about the visual object may include at least one of information about the location of the external object or information about the time at which the visual object was identified.

[0251] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, based on information about the visual object, a spatial map of external space including a first location where the wearable device is located and a second location where the external wearable device is located. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a location of the external object as a third location within the spatial map.

[0252] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a path from the first location where the wearable device is located to the third location identified as the location of the external object, based on information about the location of the external object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the element for indicating the location of the external object, based on the identified path.

[0253] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain a spatial map of the external object based on the at least one image. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify the visual object of the external object based on the spatial map.

[0254] According to one embodiment, a method of a wearable device may include an operation of identifying information about a target object. The method may include an operation of identifying a visual object related to an external object corresponding to the target object within at least one image acquired based on a gaze of a user of the wearable device during a specified time period. The method may include an operation of identifying, based on the gaze of the user, whether a first image including the visual object is displayed through a display of the wearable device. The method may include an operation of modifying the first image to emphasize the visual object based on identifying that the first image including the visual object is displayed through the display. The method may include an operation of superimposing an affordance on the second image to modify the gaze of the user to display the first image based on identifying that a second image distinct from the first image including the visual object is displayed through the display.

[0255] According to one embodiment, the information about the target object may include information about the shape and / or visual pattern of the target object and information about the size of the target object. The method may include an operation of identifying one or more candidate visual objects corresponding to the target object based on the information about the shape and / or visual pattern of the target object. The method may include an operation of identifying the visual object among the one or more candidate visual objects based on the information about the size of the target object.

[0256] In one embodiment, the method may include an operation of identifying at least one target image of the target object. The method may include an operation of identifying information regarding a shape and / or visual pattern of the target object based on the at least one target image.

[0257] According to one embodiment, the method may include an operation of requesting an external wearable device connected to the wearable device to identify the visual object. The method may include an operation of receiving information about the visual object identified through the external wearable device based on the request. The method may include an operation of displaying an element for indicating a location of the external object based on the information about the visual object.

[0258] According to one embodiment, the information about the visual object may include at least one of information about the location of the external object or information about the time at which the visual object was identified.

[0259] In one embodiment, the method may include an operation of identifying a spatial map of an external space, including a first location where the wearable device is located and a second location where the external wearable device is located, based on information about the visual object. The method may include an operation of identifying a location of the external object as a third location within the spatial map.

[0260] According to one embodiment, the method may include an operation of identifying a path from the first location where the wearable device is located to the third location identified as the location of the external object, based on information regarding the location of the external object. The method may include an operation of displaying the element for indicating the location of the external object, based on the identified path.

[0261] The terms '...part' and / or 'module' described above mean a unit that processes at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software.

[0262] According to the above-described embodiment, the wearable device can identify an external object corresponding to the target object in a real-world environment based on information regarding the shape and / or visual pattern of the target object and information regarding the size of the target object. The wearable device can highlight and display a visual object corresponding to the identified external object.

[0263] The wearable device can identify a plurality of external objects corresponding to the plurality of visual objects and positions of each of the plurality of external objects based on identifying a plurality of visual objects in at least one image acquired during a specified time period using at least one image. The wearable device can identify an external object corresponding to a target object among the plurality of external objects. The wearable device can identify the position of the external object corresponding to the target object. The wearable device can guide movement to the position of the external object corresponding to the target object.

[0264] The wearable device can identify a visual object corresponding to a target object among a plurality of external objects, primarily based on information about the shape and / or visual pattern of the target object. The wearable device can secondarily identify a visual object corresponding to the external object corresponding to the target object among the candidate visual objects, based on information about the size of the target object. The wearable device can identify a visual object corresponding to the external object corresponding to the target object by identifying an external object corresponding to the size of the target object based on depth information about the candidate visual objects acquired using a ToF sensor (or a plurality of cameras).

[0265] Electronic devices according to embodiments disclosed herein may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments disclosed herein are not limited to the aforementioned devices.

[0266] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among the phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0267] In one embodiment of this document, the term "module" used may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0268] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0269] According to one embodiment, the method according to one embodiment disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0270] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In wearable devices, display; One or more cameras; A memory including one or more storage media storing instructions; and comprising at least one processor including a processing circuit; The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Identify information about the target object; Identifying a visual object, in at least one image acquired by said one or more cameras and based on the gaze of a user of said wearable device during a specified time period, regarding an external object corresponding to said target object; Based on the gaze of the user, identifying whether a first image including the visual object is displayed through the display; Based on identifying that the first image including the visual object is displayed via the display, altering the first image to emphasize the visual object; Based on identifying that a second image distinct from the first image including the visual object is displayed through the display, causing an affordance for changing the user's gaze to display the first image to be displayed as an overlay on the second image. Wearable devices.

2. In the first paragraph, the information about the target object is, Contains information about the shape and / or visual pattern of the target object and information about the size of the target object, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on information about the shape and / or visual pattern of the target object, identifying one or more candidate visual objects corresponding to the target object; Based on information about the size of the target object, causing the visual object to be identified among the one or more candidate visual objects. Wearable devices.

3. In the second paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Identifying at least one target image of said target object; causing information about the shape and / or visual pattern of the target object to be identified based on at least one target image; Wearable devices.

4. In the third paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on user input, causing information about the size of said target object to be identified; Wearable devices.

5. In the first paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on identifying that the first image including the visual object is displayed through the display, causing display of a remaining area within the first image to be changed except for an area corresponding to the visual object. Wearable devices.

6. In the fifth paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing information about the distance between said external object and said wearable device to be displayed in association with said visual object; Wearable devices.

7. In the first paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on obtaining information about a plurality of visual objects included in at least one image, storing information about the plurality of visual objects in the memory; Setting information about the plurality of visual objects and information about the target object as input values ​​of a designated model indicated by a plurality of parameters; Based on the output value of the above specified model, causing the visual object to be identified with respect to the external object corresponding to the target object within the at least one image. Wearable devices.

8. In the first paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Acquiring an image of an external environment including the first image and the second image using one or more cameras; Causing one of the first image and the second image to be displayed through the display based on some of the images of the external environment. Wearable devices.

9. In the first paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Requesting identification of the visual object from an external wearable device connected to the wearable device; Based on the above request, receiving information about the visual object identified through the external wearable device; Causing an element to be displayed to indicate the location of the external object based on information about the visual object. Wearable devices.

10. In paragraph 9, information about the visual object is: Containing at least one of information regarding the location of the external object or information regarding the time at which the visual object was identified; Wearable devices.

11. In the 10th paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on the information about the visual object, a spatial map of an external space including a first location where the wearable device is located and a second location where the external wearable device is located is identified; causing the location of said external object to be identified as a third location within said spatial map; Wearable devices.

12. In the 11th paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on information about the location of the external object, identifying a path from the first location where the wearable device is located to the third location identified as the location of the external object; Based on the above identified path, causing the element to be displayed to indicate the location of the external object; Wearable devices.

13. In the first paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on at least one image, a spatial map of the external object is obtained; Based on the above spatial map, causing the visual object to be identified with respect to the external object, Wearable devices.

14. In a method performed in a wearable device, An action that identifies information about a target object; An action of identifying a visual object, relative to an external object corresponding to the target object, within at least one image acquired based on the gaze of the user of the wearable device during a specified time period; An action of identifying whether a first image including the visual object is displayed through a display of the wearable device based on the gaze of the user; An operation of changing the first image to emphasize the visual object based on identifying that the first image including the visual object is displayed through the display; and An operation of displaying an affordance for changing the user's gaze to display the first image as an overlay on the second image based on identifying that a second image, which is distinct from the first image including the visual object, is displayed through the display, method.

15. In paragraph 14, the information about the target object is: Contains information about the shape and / or visual pattern of the target object and information about the size of the target object, The above method, An operation of identifying one or more candidate visual objects corresponding to the target object based on information about the shape and / or visual pattern of the target object; and Further comprising an action of identifying the visual object among the one or more candidate visual objects based on information about the size of the target object. method.

Images