Wearable device for acquiring information on external space including reflector and method thereof

Abstract

A wearable device according to an embodiment may comprise: a camera; a memory for storing instructions; and a processor for executing the instructions. The processor may be configured to obtain, by using the camera, an image of an external space including the wearable device. The processor may be configured to identify, on the basis of recognizing objects in the image, a portion of the image related to a reflector that reflects light. The processor may be configured to run a software application by using information based on whether each of feature points in the image is included in the portion, so as to provide a virtual space at least partially mapped to the external space.

Description

Wearable device and method for obtaining information about an external space including a reflector

[0001] The present disclosure relates to a wearable device and method for obtaining information about an external space including a reflector.

[0002] To provide an enhanced user experience (UX), electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] According to one embodiment, a wearable device may include a camera, a memory for storing instructions, and a processor for executing the instructions. The processor may be configured to acquire an image of an external space including the wearable device using the camera. The processor may be configured to identify a portion of the image associated with a reflector that reflects light based on object recognition of the image. The processor may be configured to execute a software application using information based on whether each of the feature points of the image is included in the portion, thereby providing a virtual space at least partially mapped to the external space.

[0005] In one embodiment, a method of a wearable device may include an operation of acquiring an image of an external space including the wearable device using a camera of the wearable device. The method may include an operation of identifying a portion of the image associated with a reflector that reflects light based on object recognition of the image. The method may include an operation of executing a software application using information based on whether each of the feature points of the image is included in the portion, thereby providing a virtual space at least partially mapped to the external space.

[0006] In one embodiment, a wearable device may include a camera, a memory for storing instructions, and a processor for executing the instructions. The processor may be configured to identify an external space including the wearable device based on first feature points of an image acquired using the camera. The processor may be configured to identify second feature points, which have been identified based on the camera and have coordinate values ​​corresponding to the first external space, based on identifying the wearable device included in the first external space next to the second external space, among the first external space or a second external space including a reflector. The processor may be configured to obtain information related to the second external space using the one or more third feature points and the first feature points, based on identifying the wearable device included in the second external space, among the first external space or the second external space.

[0007] In one embodiment, a method of a wearable device may include an operation of identifying an external space including a wearable device based on first feature points of an image acquired using a camera of the wearable device. The method may include an operation of identifying second feature points, which have been identified based on the camera and have coordinate values ​​corresponding to the first external space, based on identifying the wearable device included in the first external space next to the second external space, among the first external space or a second external space including a reflector. The method may include an operation of obtaining information related to the first external space based on the first feature points and fourth feature points, which are different from one or more third feature points associated with the reflector, among the second feature points. The method may include an operation of obtaining information related to the second external space using the one or more third feature points and the first feature points, based on identifying the wearable device included in the second external space, among the first external space or the second external space.

[0008] Figure 1 illustrates an embodiment of a wearable device that obtains information about external space.

[0009] FIGS. 2A and 2B illustrate an example block diagram of a wearable device, according to one embodiment.

[0010] FIG. 3 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0011] FIG. 4 illustrates an example of the operation of a wearable device to identify a portion associated with a reflector within an image.

[0012] FIGS. 5A and 5B illustrate an example of the operation of a wearable device to identify portions associated with a reflector in each of sequentially acquired images.

[0013] FIGS. 6A, 6B and 6C illustrate an example of an operation of a wearable device to identify a location of the wearable device within an external space including a reflector.

[0014] Figure 7 illustrates an example of a user interface (UI) displayed by a wearable device that has identified a reflector.

[0015] FIG. 8 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0016] FIG. 9A illustrates an example of a perspective view of a wearable device, according to one embodiment.

[0017] FIG. 9b illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.

[0018] FIGS. 10A and 10B illustrate an example of an appearance of a wearable device according to one embodiment.

[0019] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0020] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0021] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0022] FIG. 1 illustrates an embodiment of a wearable device (101) that acquires information about an external space. The wearable device (101) may include a head-mounted display (HMD) that is wearable on the head of a user (110). The wearable device (101) may be referred to as a head-mounted device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) or visible see-through device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of a hardware configuration included in the wearable device (101) is exemplarily described with reference to FIG. 2A and / or FIG. 2B. An example of the structure of a wearable device (101) that can be worn on the head of a user (110) is described with reference to FIGS. 9A, 9B, 10A, and / or 10B. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) for attaching to the head of a user to form an HMD.

[0023] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) may see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0024] In one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include a housing that covers the eyes of the user (110). The wearable device (101), in this state, may include a display disposed on a first side of the housing facing the eyes. The wearable device (101) may include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (101) may acquire images and / or videos representing ambient light. The wearable device (101) can output the image and / or video within the display disposed on the first surface, thereby allowing the user (110) to recognize the ambient light through the display. The displaying area (or active area) of the display disposed on the first surface can be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object into the image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with a real object recognized by the ambient light.

[0025] According to one embodiment, the wearable device (101) can identify or recognize a position (or location) and / or direction (or orientation) of the wearable device (101) based on an image (and / or video) obtained or acquired using a camera. An exemplary operation of the wearable device (101) to identify the position based on an image is described with reference to FIG. 3 and / or FIG. 8. Referring to FIG. 1, exemplary images (140, 150) obtained by a camera included in the wearable device (101) are illustrated. Using a camera positioned toward one direction (e.g., the front direction) of the wearable device (101), the wearable device (101) can obtain an image of the external space. While worn by a user (110), the wearable device (101) can control the camera to acquire images (140, 150). The image (140) can be acquired while the user (110) wearing the wearable device (101) looks in the direction (dh1). The image (150) can be acquired while the user (110) wearing the wearable device (101) looks in the direction (dh2).

[0026] Referring to FIG. 1, an exemplary environment of a wearable device (101) placed in an external space including a reflector (130), such as a mirror, is illustrated. The reflector (130) may include a material (e.g., glass, acrylic, metal, and / or liquid) for reflecting ambient light. Embodiments are not limited thereto, and the reflector (130) may include an electronic device including a selfie camera, such as a smart mirror. The reflector (130) may have a relatively high reflexivity. The reflector (130) may include one surface on which the material is uniformly distributed. Within the exemplary environment of FIG. 1, incident light on the one surface of the reflector (130) may be at least partially reflected toward the wearable device (101). Referring to FIG. 1, portions of images (140, 150) corresponding to the reflector (130) can represent reflected light propagated toward the wearable device (101).

[0027] The wearable device (101) may identify one or more external objects associated with an image (e.g., images (140, 150)) based on object recognition for the image. For example, within the image (140), the wearable device (101) may identify wall surfaces (120, 122) corresponding to each of the visual objects (141, 142). Within the image (140), the wearable device (101) may identify a flowerpot (132) placed within an external space based on a visual object (144). Within the image (140), the wearable device (101) may identify a reflector (130) based on a visual object (143). Based on the segmented visual objects (141, 142, 143, 144) from the image (140), the wearable device (101) can identify locations of external objects (e.g., wall surfaces (120, 122), flower pots (132), and / or reflectors (130)). Similarly, based on the segmented visual objects (151, 152, 153, 154) from the image (150), the wearable device (101) can identify locations of external objects (e.g., wall surfaces (120, 122), flower pots (132), and / or reflectors (130)). The locations of the external objects identified by the wearable device (101) may correspond to relative locations of the external objects with respect to the wearable device (101). The operation of a wearable device (101) for identifying a portion of an image (140) related to a reflector (130), such as a visual object (143), is described with reference to FIG. 4. An example of a user interface (UI) displayed by a wearable device (101) that has identified a reflector (130) is described with reference to FIG. 7.

[0028] For example, the reflector (130) may reflect light from an external object different from the reflector (130) toward the wearable device (101). For example, a user (110) wearing the wearable device (101) adjacent to the reflector (130) may be captured in a portion of the image (140) corresponding to the reflector (130). Similarly, a user (110) wearing the wearable device (101) may be captured in a portion of the image (150) corresponding to the reflector (130). The wearable device (101) may more accurately calculate the position and / or orientation of the wearable device (101) based on the visual objects (143, 153) determined to correspond to the reflector (130) in each of the images (140, 150). For example, the wearable device (101) can identify whether each of the feature points of the image (140) corresponds to reflected light of the reflector (130). The wearable device (101) can determine whether to use each of the feature points of the image (140) to calculate the position and / or direction of the wearable device (101) based on whether each of the feature points corresponds to reflected light of the reflector (130). The operation of the wearable device (101) to determine whether each of the feature points corresponds to the reflected light is described with reference to FIGS. 5A, 5B, 6A, 6B, and / or 6C.

[0029] In one embodiment, the position and / or orientation of the wearable device (101) calculated by the wearable device (101) may be used to execute a software application (e.g., a software application for virtual reality) of the wearable device (101). For example, the wearable device (101) may execute the software application based on information indicating whether each of the feature points of the image (140) is included in a portion of the image (140) associated with the reflector (130). The wearable device (101) executing the software application may, based on the information, display a virtual space provided by the software application and at least partially mapped (or coupled to) an external space.

[0030] The operation of calculating the position and / or direction of the wearable device (101) may be referred to as head tracking (HeT) (or head tracking function) in terms of tracking the position and / or direction of the head of a user (110) wearing the wearable device (101). For example, the wearable device (101) may identify the position (ph) of the head of the user (110) and / or the wearable device (101) based on at least one of the images (140, 150). The wearable device (101) may identify the directions (dh1, dh2) of the head and / or the wearable device (101) at the respective timings at which the respective images (140, 150) were captured. While displaying a virtual space mapped to an external space, the wearable device (101) that acquired the image (140) can display one or more virtual objects set to be positioned in the direction (dh1). When the user (110) looking at the direction (dh1) turns his / her head toward the direction (dh2), the wearable device (101) can display another virtual object set to be positioned in the direction (dh2) based on the acquisition of the image (150). Based on head tracking, the wearable device (101) can display a screen (e.g., a screen based on AR and / or VR) synchronized with the motion of the user's (110) head.

[0031] As described above, according to one embodiment, the wearable device (101) can identify at least a portion of the image associated with a reflector (130) that reflects light based on object recognition of the image (e.g., images (140, 150)). For example, the wearable device (101) can obtain information including at least one of a shape, a position, or a size of the portion within the image. At least a portion of the image associated with the reflector (130) can be referred to as a reflector region. The wearable device (101) can more accurately calculate a position and / or direction of the wearable device (101) based on a positional relationship between feature points of the image and the reflector region.

[0032] Hereinafter, with reference to FIG. 2a and / or FIG. 2b, an example of a hardware configuration included in a wearable device (101) for head tracking is described.

[0033] FIGS. 2A and 2B illustrate an example of a block diagram of a wearable device (101), according to one embodiment. The wearable device (101) described with reference to FIG. 1 may include at least one of the hardware components of the wearable device (101) distinguished by different blocks in FIGS. 2A and / or 2B.

[0034] Referring to FIG. 2A, a wearable device (101) according to one embodiment may include at least one of a processor (210), a memory (215), a display (220), a camera (225), a sensor (230), and a communication circuit (235). The processor (210), the memory (215), the display (220), the camera (225), the sensor (230), and / or the communication circuit (235) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (202). The type and / or number of hardware components included in the wearable device (101) is not limited to those illustrated in FIG. 2A. For example, the wearable device (101) may include only some of the hardware components illustrated in FIG. 2A.

[0035] According to one embodiment, the processor (210) of the wearable device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. The processor (210) may have a multi-core processor structure, such as a dual core, a quad core, a hexa core, and / or an octa core.

[0036] According to one embodiment, the memory (215) of the wearable device (101) may include a hardware component for storing data and / or instructions input and / or output to the processor (210). The memory (215) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multimedia card (eMMC). In one embodiment, the memory (215) may be referred to as storage.

[0037] In one embodiment, the display (220) of the wearable device (101) can output visualized information (e.g., the screen (710) of FIG. 7) to a user of the wearable device (101) (e.g., the user (110) of FIG. 1). For example, the display (220) can be controlled by a processor (210) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (220) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). Embodiments are not limited thereto, and for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (220) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (220) may be referred to as a display panel and / or a display module.

[0038] In one embodiment, the camera (225) of the wearable device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The camera (225) may be referred to as an image sensor and may be included in the sensor (230) of FIG. 2A. The plurality of optical sensors included in the camera (225) may be arranged in the form of a two-dimensional array. The camera (225) may acquire electrical signals from each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (225) may mean one (a) two-dimensional frame data acquired from the camera (225). For example, video data captured using a camera (225) may mean a sequence of a plurality of two-dimensional frame data obtained from the camera (225) according to a frame rate. The camera (225) may be positioned toward the direction in which the camera (225) receives light and may further include a flash light for outputting light toward the direction.

[0039] According to one embodiment, the wearable device (101) may include a plurality of cameras, for example, cameras (225), which are arranged facing different directions. Referring to FIG. 2A, the camera (225) included in the wearable device (101) may include a gaze tracking camera (225-1) and / or an outward camera (225-2). The gaze tracking camera (225-1) may be arranged toward at least one of the two eyes of a user wearing the wearable device (101). The processor (210) may identify the direction of the user's gaze using images and / or videos acquired from the gaze tracking camera (225-1). The gaze tracking camera (225-1) may include an infrared (IR) sensor. The gaze tracking camera (225-1) may be referred to as an eye sensor, a gaze tracker, and / or an eye tracker.

[0040] In one embodiment, the external camera (225-2) may be positioned facing the front of a user wearing the wearable device (101) (e.g., the direction in which both eyes may face). The direction in which the external camera (225-2) faces is not limited to the front. The external camera (225-2) may be positioned on any one of the surfaces different from the surface of the wearable device (101) that is in contact with the user while the device is worn by the user. Using the images and / or videos acquired from the external camera (225-2), the processor (210) may identify an external object (e.g., the reflector (130) of FIG. 1). The embodiment is not limited thereto, and the processor (210) may identify the position, shape, and / or gesture (e.g., hand gesture) of a hand based on the images and / or videos acquired from the external camera (225-2).

[0041] According to one embodiment, a sensor (230) of a wearable device (101) may generate electrical information from non-electronic information related to the wearable device (101), which may be processed and / or stored by a processor (210) and / or a memory (215) of the wearable device (101). The information may be referred to as sensor data. The sensor (230) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor (or ToF camera) for detecting a geographic location of the wearable device (101).

[0042] In one embodiment, the sensor (230) may include an inertial measurement unit (IMU) for detecting physical motion of the wearable device (101). An acceleration sensor, a geomagnetic sensor, a gyro sensor, or any combination thereof may be referred to as an IMU. The acceleration sensor may output an electrical signal representing gravitational acceleration and / or acceleration along each of a plurality of axes (e.g., the x-axis, the y-axis, and the z-axis) that are perpendicular to each other and based on a designated origin of the wearable device (101). The gyro sensor may output an electrical signal representing an angular velocity of each of the plurality of axes. The gyro sensor may be referred to as an angular velocity sensor. The geomagnetic sensor may output an electrical signal representing the magnitude of a magnetic field formed in the wearable device (101) along each of the plurality of axes (e.g., the x-axis, the y-axis, and / or the z-axis). For example, an acceleration sensor, a gyro sensor, and / or a geomagnetic sensor may repeatedly output sensor data including accelerations, angular velocities, and / or magnitudes of magnetic fields of a number of axes based on a specified period (e.g., 1 millisecond).

[0043] In one embodiment, the communication circuit (235) of the wearable device (101) may include circuitry for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device. The communication circuit (235) may include, for example, at least one of a modem (modulator and demodulator), an antenna, and an optical / electronic (O / E) converter. The communication circuit (235) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G new radio (NR), 6G and / or above-6G. In one embodiment, the communication circuit (235) may be referred to as a communication processor and / or a communication module.

[0044] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (210) of the wearable device (101) may be stored in the memory (215) of the wearable device (101). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (101) and / or the processor (210) may perform at least one of the operations of FIG. 3 and / or FIG. 8 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (101) may mean that one or more instructions provided in the form of an application are stored in a memory (215), and that the one or more applications are stored in a format executable by the processor (210) (e.g., a file having an extension specified by the operating system of the wearable device (101)). For example, an application may include a program and / or a library related to a service provided to a user.

[0045] Referring to FIG. 2A, programs installed in the wearable device (101) may be classified into any one of different layers, including an application layer (240), a framework layer (250), and / or a hardware abstraction layer (HAL) (280), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (101) (e.g., a display (220), a camera (225), a sensor (230), and / or a communication circuit (235)) may be classified within the hardware abstraction layer (280). The framework layer (250) may be referred to as an XR framework layer in terms of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 2a may be logically (or for convenience of explanation) separated, but this does not mean that the address space of the memory (215) is separated by the layers.

[0046] For example, within the framework layer (250), programs designed to target at least one of the hardware abstraction layer (280) and / or the application layer (240) (e.g., a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye-gaze tracker (274), and / or a face tracker (275)) may be classified. Programs classified within the framework layer (250) may provide an executable API (application programming interface) based on other programs.

[0047] For example, within the application layer (240), programs designed to target users of the wearable device (101) may be classified. As examples of programs classified into the application layer (240), an extended reality (XR) system user interface (UI) (241) and / or an XR application (242) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) classified into the application layer (240) may call an application programming interface (API) to cause execution of functions supported by programs classified into the framework layer (250).

[0048] For example, the wearable device (101) may display one or more visual objects on the display (220) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (241). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user with functions available within the virtual space based on the execution of the XR system UI (241).

[0049] Referring to FIG. 2A, a lightweight renderer (243) and / or an XR plug-in (244) are illustrated to be included within the XR system UI (241), but are not limited thereto. For example, based on the XR system UI (241), the processor (210) may execute a lightweight renderer (243) and / or an XR plug-in (244) within the framework layer (250).

[0050] For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (243). The lightweight renderer (243) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (243) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (244). The XR plugin (244) can be referred to as an open XR native client from the perspective of defining (or setting up) the entire rendering pipeline.

[0051] For example, the wearable device (101) may display a screen representing at least a portion of a virtual space on the display (220) based on the execution of the XR application (242). The XR plug-in (244-1) included in the XR application (242) may include instructions that support functions similar to those of the XR plug-in (244) of the XR system UI (241). Descriptions of the XR plug-in (244-1) that overlap with those of the XR plug-in (244) may be omitted. The wearable device (101) may cause the execution of the virtual space manager (251) based on the execution of the XR application (242).

[0052] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the virtual space manager (251). For example, the virtual space manager (251) may include a platform for supporting the virtual space service. The wearable device (101) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (230) based on the execution of the virtual space manager (251), and may display at least a portion of the virtual space on the display (220). The virtual space manager (251) may be referred to as a composition presentation manager (CPM).

[0053] For example, the virtual space manager (251) may include a runtime service (252). As an example, the runtime service (252) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (252). As an example, the wearable device (101) may perform rendering for a virtual space service to the user based on the execution of the runtime service (252). For example, a function related to a virtual space, executable by the application layer (240), may be supported based on the execution of the runtime service (252).

[0054] For example, the virtual space manager (251) may include a pass-through manager (253). Based on the execution of the pass-through manager (253), the wearable device (101) may display a screen representing a virtual space on the display (220), while superimposing another screen representing an actual space acquired through an external camera (225-2) on at least a portion of the screen.

[0055] For example, the virtual space manager (251) may include an input manager (254). The wearable device (101) may identify data (e.g., sensor data) obtained by executing one or more programs included in the recognition service layer (270) based on the execution of the input manager (254). The wearable device (101) may use the obtained data to identify user input related to the wearable device (101). The user input may be related to a motion (e.g., hand gesture), gaze, and / or speech of the user identified by the sensor (230).

[0056] For example, the perception abstract layer (260) can be used for data exchange between the virtual space manager (251) and the perception service layer (270). From the perspective of being used for data exchange between the virtual space manager (251) and the perception service layer (270), the perception abstract layer (260) can be referred to as an interface. For example, the perception abstract layer (260) can be referenced as OpenPX. The perception abstract layer (260) can be used for a perception client and a perception service.

[0057] According to one embodiment, the recognition service layer (270) may include one or more programs for processing data acquired from a sensor (230) and / or a camera (225). The one or more programs may include at least one of a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye tracker (274), and / or a face tracker (275). The type and / or number of the one or more programs included in the recognition service layer (270) are not limited to those illustrated in FIG. 2A.

[0058] For example, the wearable device (101) can identify the pose of the wearable device (101) using the sensor (230) based on the execution of the position tracker (271). The wearable device (101) can identify the 6 degrees of freedom pose (6 DoF pose) of the wearable device (101) using data acquired using the external camera (225-2) and / or IMU (e.g., a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (271). The position tracker (271) may be referred to as a head tracking (HeT) module (or head tracker, a head tracking program).

[0059] For example, the wearable device (101) may obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of the space recognizer (272). The wearable device (101) may reconstruct the surrounding environment of the wearable device (101) in three dimensions using data obtained using the external camera (225-2) based on the execution of the space recognizer (272). The wearable device (101) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (101) reconstructed in three dimensions based on the execution of the space recognizer (272). The space recognizer (272) may be referred to as a scene understanding (SU) module (or a scene understanding program).

[0060] For example, the wearable device (101) can identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (101) based on the execution of the gesture tracker (273). As an example, the wearable device (101) can identify a pose and / or gesture of a hand of a user using data acquired from an external camera (225-2) based on the execution of the gesture tracker (273). As an example, the wearable device (101) can identify a pose and / or gesture of a hand of a user based on data (or images) acquired using the external camera (225-2) based on the execution of the gesture tracker (273). The gesture tracker (273) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0061] For example, the wearable device (101) can identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (274). As an example, the wearable device (101) can identify eye movements of the user using data acquired from the gaze tracking camera (225-1) based on the execution of the gaze tracker (274). The gaze tracker (274) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0062] For example, the recognition service layer (270) of the wearable device (101) may further include a face tracker (275) for tracking the user's face. For example, the wearable device (103) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (275). The wearable device (101) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (275). As an example, the wearable device (101) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a sensor (230) (e.g., an image sensor directed at at least a portion of the user's face) based on the execution of the face tracker (275).

[0063] According to one embodiment, the wearable device (101) can identify a location of the wearable device (101) within an external space, as indicated by information obtained from a camera (225) and / or a sensor (230). Based on the location, the wearable device (101) can provide at least a portion of a virtual space mapped (or linked) with the external space. For example, the wearable device (101) can determine a location of the wearable device (101) within the virtual space based on the location. The determined location within the virtual space can correspond to a reference point (e.g., a view point) that indicates a portion of the virtual space to be displayed through the display (220). The processor (210) of the wearable device (101) can execute one or more instructions to identify the location within the external space by executing the location tracker (271) of FIG. 2A.

[0064] Referring to FIG. 2b, instructions included in the position tracker (271) may be divided into a sensor manager (291), a position and direction processor (293), a feature point extractor (294), a reflector detector (295), and / or a map generator (296) based on their functions. The instructions may be executed by the processor (210) of the wearable device (101).

[0065] Based on the execution of the sensor manager (291), the wearable device (101) can store the sensor data of the sensor (230) in the memory (215) of the wearable device (101). Based on the execution of the sensor manager (291), the wearable device (101) can manage the sensor data stored in the memory (215). For example, the wearable device (101) can store and / or delete sensor data (e.g., angular velocity, acceleration, and / or direction of a magnetic field) acquired from an IMU in the memory (215). The wearable device (101) can acquire a value that integrates the angular velocity, acceleration, and / or direction of a magnetic field included in the sensor data acquired from the IMU. The value acquired by the wearable device (101) can indicate the direction and / or position of the wearable device (101) in external space. The wearable device (101) can accumulate images and / or sensor data accumulated over a specific time period to obtain vector data corresponding to the time period. The vector data can be obtained to reduce the amount of calculation required to identify the position and / or direction of the wearable device (101). The wearable device (101) can obtain the vector data in response to a call to an API for accumulating sensor data. The embodiment is not limited thereto, and the wearable device (101) can store images and / or sensor data accumulated over a specific time period in the memory (215) of the wearable device (101).

[0066] Based on the execution of the feature point extractor (294), the wearable device (101) can identify one or more feature points from an image (e.g., a frame image of a video) acquired from the camera (225) (e.g., images (140, 150) of FIG. 1). A feature point (or keypoint) of an image may be a point within the image that is used to identify a feature of the image (or an object captured by the image). For example, by comparing feature points included in each of different images, the wearable device (101) can identify an external object commonly captured in the images. By comparing feature points of different images, the wearable device (101) can identify a landmark of an external object commonly captured in the images. Based on the feature points included in the images, the wearable device (101) can calculate three-dimensional spatial coordinates of the external object associated with the feature points. Based on the feature points included in the image, the wearable device (101) can determine the image as a key frame.

[0067] Based on the execution of the feature point extractor (294), the wearable device (101) can perform calculations represented by algorithms such as SIFT (scale invariant feature transform), Harris corner detection, FAST (features from accelerated segment test), and / or ORB (oriented fast and rotated brief). Based on the calculations, the wearable device (101) can extract feature points from the images, or perform comparison (or matching) of feature points included in the images. By comparing feature points included in images acquired at different timings and / or locations, the wearable device (101) can determine three-dimensional coordinates of each of the feature points. The wearable device (101) that identifies a feature point commonly included in a plurality of images can obtain information indicating timings (or the plurality of images) at which the feature point was identified.

[0068] Based on the execution of the feature point extractor (294), the wearable device (101) can acquire one or more feature points used to calculate the position and / or orientation of the wearable device (101) from an image acquired from the camera (225). The wearable device (101) that has acquired a plurality of images can select and / or determine a key frame from among the plurality of images based on the number of feature points extracted from each of the plurality of images. For example, an image in which a relatively large number (or a number exceeding a specified number) of feature points are detected can be selected as a key frame. For example, the wearable device (101) can compare feature points commonly included in different images and select an image in which the number of feature points in common with other images acquired at a timing prior to a specific timing at which the image was identified as a key frame is relatively small. The wearable device (101) can store information (292) indicating the result of selecting feature points and / or key frames extracted from images in the memory (215) of the wearable device (101).

[0069] Based on the execution of the reflector detector (295), the wearable device (101) can identify at least a portion corresponding to a reflector (e.g., reflector (130) of FIG. 1) within the image. Having identified the at least a portion corresponding to the reflector, the wearable device (101) can calculate a position and / or a size of the at least a portion within the image. Based on the at least a portion, the wearable device (101) can calculate a position (e.g., a position relative to the wearable device (101)) and / or a size of the reflector.

[0070] In one embodiment, the wearable device (101) may execute a computational model for extracting a reflector from an image based on a reflector detector (295). The computational model may include an artificial neural network that simulates human reasoning activities. For example, the computational model may include an artificial neural network for processing an image, which is a group of pixels arranged in two dimensions, such as a convolutional neural network (CNN). The embodiment is not limited thereto, and the computational model may include a feedforward neural network (FNN), a long-short term memory (LSTM), and / or a recurrent neural network (RNN). The artificial neural network simulated based on the reflector detector (295) may be a model that has been pre-trained based on supervised learning and / or unsupervised learning.

[0071] Based on the execution of the reflector detector (295), the wearable device (101) can obtain information (e.g., a probability map) including probabilities that each pixel included in the image corresponds to a reflector. Based on the information, the wearable device (101) can identify a portion of the image (e.g., a portion having a shape of a square and / or a circle) corresponding to a reflector. Based on the execution of the feature point extractor (294), the wearable device (101) that has identified one or more feature points from the image can determine whether each of the one or more feature points corresponds to a reflector based on the probabilities acquired by the reflector detector (295). The wearable device (101) can store, in information (292) stored in the memory, parameters (e.g., flags and / or probabilities) indicating whether each of the feature points included in the information (292) corresponds to a reflector.

[0072] Based on the execution of the map generator (296), the wearable device (101) can generate map information for the external space in which the wearable device (101) is located using information (292) representing feature points and / or key frames. The wearable device (101) can generate map information (e.g., a local map) for each of different external spaces having a size less than a specified size. The wearable device (101) can merge the map information for each of the external spaces. By comparing an image acquired in real time using the camera (225) and the map information generated by the map generator (296), the wearable device (101) can determine whether the current location of the wearable device (101) corresponds to an external space and / or point corresponding to the map information. If the current location corresponds to a point corresponding to the map information, the wearable device (101) can identify a revisitation of the wearable device (101) to the point.

[0073] Map information (e.g., local map) generated based on the execution of the map generator (296) may include information on feature points and / or key frames used to generate the map information. Map information generated based on the execution of the map generator (296) may be stored in the memory (215) of the wearable device (101). Based on the execution of the map generator (296), the wearable device (101) may manage (e.g., create and / or delete) the map information stored in the memory (215). In one embodiment, the map information may be a unit that stores feature points and / or key frames obtained (or extracted) based on the feature point extractor (294). The wearable device (101) may compare different local maps to determine whether to merge the local maps. For example, if the similarity between local maps exceeds a specified threshold, the wearable device (101) may merge the local maps. Merging the local maps by the wearable device (101) may mean managing (e.g., transmitting, duplicating, and / or deleting) the feature points and / or key frames corresponding to each of the local maps as a single unit.

[0074] Based on the execution of the map generator (296), the wearable device (101) can determine the current location of the wearable device (101) from an image identified in real time from the camera (225). For example, if feature points corresponding to specific map information generated by the map generator (296) are identified, the wearable device (101) can determine the location corresponding to the specific map information as the current location of the wearable device (101). For example, if an image similar to a key frame stored by the map generator (296) is acquired from the camera (225), the wearable device (101) can determine the location corresponding to the key frame as the current location of the wearable device (101). A local map that is separated from the current location can be removed from the volatile memory of the wearable device (101) or can be stored in a non-volatile memory.

[0075] Based on the execution of the position and direction processor (293), the processor (210) of the wearable device (101) can calculate the position and / or direction of the wearable device (101). The wearable device (101) can calculate the position and / or direction of the wearable device (101) based on at least one of information (292) including feature points and / or key frames and / or information acquired by the reflector detector (295). For example, the wearable device (101) can calculate the position and / or direction of the wearable device (101) using information acquired by the position tracker (271). For example, based on the probabilities that each of the feature points obtained from the reflector detector (295) corresponds to a reflector, the wearable device (101) may determine whether to use the feature points to calculate the position and / or orientation of the wearable device (101). The wearable device (101) may update information (292) including the feature points and key frames based on the position and / or orientation of the wearable device (101). For example, the wearable device (101) may cause the feature points corresponding to the reflector to not be used by the map generator (296) (e.g., not be used for generating map information).

[0076] Based on the execution of the location tracker (271), the wearable device (101) can obtain information used for executing a software application (e.g., a software application included in the application layer (240)). For example, the wearable device (101) that has identified a designated API included in the software application can execute the location tracker (271) to calculate the location and / or orientation of the wearable device (101) requested by the API. The API may include parameters for requesting the location and / or orientation of the wearable device (101) at a specific timing. The software application may include instructions for repeatedly calling the designated API based on a designated cycle and / or frequency.

[0077] Based on a request called or invoked by a software application, the wearable device (101) can execute the location tracker (271). The request can be identified by an API. In order to identify the position and / or orientation of the wearable device (101) at a first timing corresponding to the request, the wearable device (101) can obtain information indicating the position and / or orientation at a second timing prior to the first timing. By combining the position and / or orientation at the second timing with sensor data (e.g., sensor data acquired from the sensor (230)) and images (e.g., images acquired from the camera (225)) in the time period between the second timing and the first timing, the wearable device (101) can calculate the position and / or orientation of the wearable device (101) at the first timing. The wearable device (101) can call a function (or callback function, routine, sub-routine and / or method) of the software application indicated by the request based on the calculated position and / or direction.

[0078] In one embodiment, the wearable device (101) may execute a location tracker (271) based on data indicating a specific timing (e.g., a timestamp) to calculate the location and / or orientation of the wearable device (101) at that timing. The wearable device (101) may calculate the location and / or orientation of the wearable device (101) based on sensor data accumulated within a time interval that includes the specific timing.

[0079] The information obtained based on the execution of the location tracker (271) and used for the execution of the software application is not limited to the position and / or orientation of the wearable device (101) described above. The wearable device (101) may execute the software application based on the map information identified based on the map generator (296) to provide a virtual space mapped to the external space. The map information may include a point cloud including one or more feature points identified from an image.

[0080] As described above, according to one embodiment, the wearable device (101) may include system software (e.g., a position tracker (271)) for obtaining information indicating a position and / or direction of the wearable device (101) as information used for executing a software application of the wearable device (101). For example, the wearable device (101) may execute instructions of the software application for reproducing a virtual space based on the information, and display at least a portion of the virtual space linked to the external space on the display (220). The wearable device (101) may execute the position tracker (271) including a reflector detector (295) to identify a reflector (e.g., a reflector (130) of FIG. 1) included in the external space. The result of identifying the reflector can be used to compensate for errors in the position and / or orientation of the wearable device (101) caused by the recognition of an image including the reflector. Hereinafter, with reference to FIG. 3, an exemplary operation of the wearable device (101) based on the identification of the reflector is described.

[0081] FIG. 3 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operations described with reference to FIG. 3. At least one of the operations of FIG. 3 may be related to an operation of a wearable device executing a location tracker (271) of FIGS. 2A and / or 2B.

[0082] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0083] Referring to FIG. 3, in operation (310), a processor of a wearable device according to an embodiment may execute a head tracking function while the wearable device is being worn by a user. The processor may switch the state of the position tracker (271) of FIG. 2A and / or FIG. 2B between an idle state and an active state based on whether the wearable device is being worn by a user (e.g., the user (110) of FIG. 1). For example, a processor that identifies motion of the wearable device based on a sensor (e.g., sensor (230) of FIG. 2A) may switch the state of the position tracker (271) from an idle state to an active state. The embodiment is not limited thereto, and a wearable device including attachable straps may identify the connection of the straps based on a switching circuit included in the straps. The wearable device that has identified the connection of the straps may perform operation (310). For example, when worn by a user, the processor can initiate execution of the head tracking function of the operation (310) by switching the state of the position tracker (271) to an active state.

[0084] Referring to FIG. 3, in operation (320), a processor of a wearable device according to one embodiment may detect a reflector (e.g., reflector (130) of FIG. 1) based on an image of a camera. The camera of operation (320) may include camera (225) of FIG. 2A and / or an external camera (225-2). The processor may perform operation (320) based on the execution of the reflector detector (295) of FIG. 2B. The processor may perform operation (320) based on a specified period or frequency. Operation (320) may be performed repeatedly (or periodically) based on a period and / or duration that exceeds, for example, a duration required to identify a reflector from a particular image. For example, the period at which operation (320) is repeatedly performed may be longer than the frame rate of the camera.

[0085] In one embodiment, the processor may repeatedly perform operation (320) based on a dynamically changing period. For example, the processor may perform operation (320) on an image selected as a key frame at a timing when a key frame is selected from among images sequentially output from a camera in a time domain. For example, the processor may perform operation (320) based on the number of feature points in the image. For example, the period for repeatedly performing operation (320) may be inversely proportional to the number of feature points extracted from the image. For example, the period for repeatedly performing operation (320) may change based on whether a reflector is detected based on the image. If a reflector is not detected from the image for a specified period, the processor may perform operation (320) based on a relatively long period. If a reflector is identified, the processor may reduce the period for repeatedly performing operation (320).

[0086] Referring to FIG. 3, in operation (330), according to one embodiment, a processor of a wearable device may obtain information related to a portion of an image associated with a reflector. The processor, which has detected a reflector based on operation (320), may perform operation (330). For example, the processor may obtain information about a portion (or area) associated with the reflector within the image. The information may include at least one of a timing of identifying the reflector (e.g., a timestamp), a type of reflector identified based on the image (e.g., a class and / or category), a size of the reflector, a location corresponding to the reflector within the image (e.g., coordinates expressed in pixels), or an identifier (e.g., an ID) for tracking the reflector. The information of operation (330) may include a pixelwise value (e.g., a flag and / or probability value) representing at least a portion associated with the reflector within the image.

[0087] Referring to FIG. 3, in operation (340), according to one embodiment, a processor of a wearable device may adjust attributes of feature points extracted from an image based on information. The feature points of operation (340) may be acquired based on the execution of a feature point extractor (294). The attribute may include a value (e.g., a flag and / or a probability value) indicating whether the corresponding feature point corresponds to the reflected light of a reflector. The image of operation (340) may correspond to the image of operation (330). The embodiment is not limited thereto, and the image of operation (340) may correspond to another image different from the image of operation (330).

[0088] For example, a processor that sequentially acquires multiple images in the time domain from a camera can extract feature points from an image at a first timing. A second timing at which the extraction of the feature points from the image is completed may be different from a third timing at which a portion related to a reflector is identified from the image. For example, the third timing may be a timing after the second timing. At the third timing, the processor can extract feature points from another image different from the image at the first timing. In one embodiment, the processor can identify or retrieve an image corresponding to the information of operation (330) and / or feature points within the image based on a timing of the image corresponding to the information included in the information of operation (330). The processor can change an attribute of each of the retrieved feature points based on whether the corresponding feature point is related to a reflector. The attribute may be included in information (292) of FIG. 2B.

[0089] Referring to FIG. 3, in operation (350), a processor of a wearable device according to one embodiment may determine a position and / or orientation of the wearable device based on attributes of feature points. The processor may perform operation (350) based on the execution of the position and orientation processor (293) of FIG. 2B. The processor may perform operation (350) using information stored in a memory (e.g., information related to feature points and / or key frames of FIG. 2B).

[0090] In one embodiment, the processor may perform operation (350) based on the execution of the map generator (296) of FIG. 2B. For example, based on the execution of the map generator (296), the processor may determine a revisit. The revisit may be determined based on whether the current location of the wearable device identified by the processor matches the location of map information acquired by the map generator (296). To determine the revisit, the processor may utilize the remaining feature points, excluding one or more feature points having a reflector-related attribute among the plurality of feature points. Based on operation (350), the processor may perform an algorithm for determining a revisit, such as loop closing. The processor performing the loop closing may not utilize at least one feature point that appears to be associated with a reflector based on operation (340) for the loop closing.

[0091] The position and / or direction of the wearable device, determined based on the action (350), may be provided to a software application executed by the wearable device (101). The processor may provide the position and / or direction of the action (350) to the software application. The processor may execute the software application based on the position and / or the direction to display at least a portion of the virtual space associated with the position and / or the direction. Since the position and / or direction of the wearable device is determined based on feature points having properties adjusted based on a reflector, the processor may more accurately determine the position and / or the direction.

[0092] Below, with reference to FIG. 4, an exemplary operation of a wearable device for detecting a reflector from an image based on an operation (320) is described.

[0093] FIG. 4 illustrates an example of an operation of a wearable device (101) to identify a portion (430) associated with a reflector within an image (140). The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operation of the wearable device (101) described with reference to FIG. 4.

[0094] According to one embodiment, the wearable device (101) can acquire an image (140) from a camera (225) (e.g., an external camera (225-2) of FIG. 2A). The wearable device (101) can execute a function for processing the image (140) based on a location tracker (271). For example, the wearable device (101) can perform an operation for identifying a portion (430) of the image (140) corresponding to a reflector based on a reflector detector (295).

[0095] Referring to FIG. 4, a neural network (410) for detecting a reflector may be provided as a part of a position tracker (271). The neural network (410) may include a set of a plurality of parameters for defining nodes included in a plurality of layers of an artificial neural network and weights (or filters) between the nodes. The wearable device (101) may include software, hardware, or a combination thereof for driving the neural network (410). The software for driving the neural network (410) may include one or more programs for performing calculations related to the plurality of parameters and / or a set of instructions (e.g., a library) called by the one or more programs. The hardware for driving the neural network (410) may include a CPU, a GPU, an NPU (neural processing unit), or a combination thereof.

[0096] Referring to FIG. 4, using a neural network (410) into which an image (140) is input, the wearable device (101) can identify the location of a portion (430) related to a reflector within the image (140). From the neural network (410) into which the image (140) is input, the wearable device (101) can obtain reflector information (420). The reflector information (420) can include data indicating the location and / or size of a portion (430) corresponding to the reflector within the image (140). Referring to FIG. 4, within an image (140) including a visual object (143) corresponding to a reflector (e.g., the reflector (130) of FIG. 1), the wearable device (101) can identify a portion (430) corresponding to one side of the reflector. In a portion (430) of the image (140), reflected light reflected from the reflector may be captured. The portion (430) within the image (140) identified based on the neural network (410) may correspond to or include a material (e.g., glass, metal, and / or an organic compound such as acrylic) enclosed by the frame of the mirror in the visual object (143) corresponding to the mirror.

[0097] The reflector information (420) generated based on the execution of the reflector detector (295) may be stored in the memory of the wearable device (101) (e.g., the memory (215) of FIG. 2A). The reflector information (420) may be stored in conjunction with the image (140). The reflector information (420) may be used to determine whether feature points included in the image (140) are related to a reflector. For example, among the feature points included in the image (140), one or more feature points included in a portion (430) identified by the reflector information (420) may have an attribute indicating that they are related to a reflector. In the above example, feature points included in another portion of the image (140) that is distinct from the portion (430) may have an attribute indicating that they are not related to a reflector (or an attribute indicating that they are related to an object different from the reflector).

[0098] In one embodiment, the size and / or location of a portion (430) indicated by the reflector information (420) may be used to determine whether feature points of the image (140) included in the portion (430) are used to identify the location of the wearable device (101). For example, if the size ratio of the portion (430) within the image (140) exceeds a specified ratio, the wearable device (101) may use the feature points located in the portion (430) to calculate the location of the wearable device (101). For example, if the size ratio of the portion (430) within the image (140) is less than a specified ratio, the wearable device (101) may discard or not use the feature points located in the portion (430) to calculate the location of the wearable device (101).

[0099] As described above, according to one embodiment, the wearable device (101) can identify a portion (430) related to a reflector from an image (140) using a neural network (410) provided with a location tracker (271). The reflector information (420) acquired by the wearable device (101) can include data indicating the size, position, and / or shape of the portion (430). The wearable device (101) that has identified the portion (430) can acquire information indicating whether each of the feature points included in the image (140) is included in the portion (430) (e.g., an attribute of the operation (340) of FIG. 3). The wearable device (101) can use the reflector information (420) corresponding to the image (140) for head tracking based on the image (140).

[0100] Hereinafter, with reference to FIG. 5a and / or FIG. 5b, an exemplary operation of a wearable device (101) performing head tracking based on images acquired sequentially within a time domain is described.

[0101] FIGS. 5A and 5B illustrate an example of an operation of a wearable device (101) for identifying portions related to a reflector in each of sequentially acquired images (521, 522, 523, 524). The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operation of the wearable device (101) described with reference to FIGS. 5A and / or 5B.

[0102] Referring to FIGS. 5A and / or 5B , an exemplary environment of a wearable device (101) worn by a user (110) adjacent to a reflector (510) (e.g., a mirror having a rectangular shape) is illustrated. Referring to the exemplary case of FIG. 5A , directions (dh1, dh2, dh3, dh4) viewed by a user (110) moving along a path connecting locations (ph1, ph2, ph3, ph4) from each location (ph1, ph2, ph3, ph4) are illustrated. Referring to FIG. 5a, images (521, 522, 523, 524) acquired from a camera of a wearable device (101) (e.g., camera (225) of FIG. 2a and / or external camera (225-2)) are shown while the direction of the head is gradually changed from direction (dh1) to direction (dh4).

[0103] An image (521) acquired at a timing t1 when a user (110) at a location (ph1) looks in a direction (dh1) may include a portion (512) corresponding to a reflector (510). The wearable device (101) may extract one or more feature points from the image (521). At a timing t1 + a, an operation of acquiring one or more feature points corresponding to the image (521) may be completed. The image (531) of FIG. 5A may be an exemplary image that displays one or more feature points on the image (521) to explain one or more feature points (e.g., marks in the shape of a square and / or triangle) extracted from the image (521).

[0104] An image (522) acquired at timing t2 when a user (110) at a location (ph2) looks in a direction (dh2) may include a portion (512) corresponding to a reflector (510). The shape of the portion (512) may be different from the portion (512) of the image (521) acquired at timing t1. At timing t2 + b after timing t2, the wearable device (101) may acquire feature points from the image (522). The image (532) may be an exemplary image for explaining the relationship between the image (522) and the feature points acquired from the image (522).

[0105] The image (523) acquired at the timing t3 when the user (110) at the location (ph3) looks in the direction (dh3) may include a portion (512) corresponding to the reflector (510). At the timing t3 + c after the timing t3, the wearable device (101) may acquire feature points from the image (523). The image (533) may be an exemplary image for explaining the relationship between the image (523) and the feature points acquired from the image (523).

[0106] The image (524) acquired at the timing t4 when the user (110) at the location (ph4) looks in the direction (dh4) may include a portion (512) corresponding to the reflector (510). At the timing t4 + d after the timing t4, the wearable device (101) may acquire feature points from the image (524). The image (534) may be an exemplary image for explaining the relationship between the image (524) and the feature points acquired from the image (524).

[0107] The wearable device (101) can identify at least one feature point commonly included in different images (531, 532, 533, 534) by comparing the feature points depicted based on the images (531, 532, 533, 534). For example, the wearable device (101) can compare the feature points of each of the images (531, 532) to confirm or identify the feature point included in all of the images (531, 532). The wearable device (101) that has identified the feature point commonly included in all of the images (531, 532) can calculate three-dimensional space coordinates corresponding to the feature point based on changes in the positions of the feature point within the images (531, 532). Based on the above three-dimensional space coordinates (or changes in the above three-dimensional space coordinates), the wearable device (101) can calculate or identify the current position and / or current direction of the wearable device (101).

[0108] The timing at which the wearable device (101) extracts feature points from a particular image (e.g., one of the images (521, 522, 523, 524)) and the timing at which the wearable device (101) identifies a portion associated with a reflector may be independent of each other. Referring to FIGS. 5A and / or 5B , a period (e.g., a, b, c, d) for extracting feature points from each of the images (521, 522, 523, 524) may be shorter than an interval (e.g., t2 - t1) between the timings at which each of the images (521, 522, 523, 524) is acquired.

[0109] Referring to FIG. 5B, at the timing t1 + x when the extraction of feature points for images (521, 522, 523) at the timings t1, t2, and t3 is completed, the wearable device (101) can identify a portion (512) related to a reflector (510) in the image (521) at the timing t1. For example, the wearable device (101) can identify, among the feature points of the image (521), an area (551) surrounding feature points corresponding to the portion (512). Among the feature points of the image (521), feature points related to the reflector can be arranged inside the area (551). The area (551) can have a polygonal shape that connects feature points (e.g., f1 feature point) adjacent to the portion (512) related to the reflector (510).

[0110] Referring to FIG. 5B, a period (e.g., x) for identifying a portion (512) related to a reflector within an image (521) may be longer than an interval (e.g., t2 - t1) between the timings at which each of the images (521, 522, 523, 524) is acquired. For example, the wearable device (101) may acquire images (522, 523) subsequent to the image (521) before the timing t1 + x at which the wearable device (101) identifies a portion related to a reflector from the image (521). Since the reflector detection for the image (521) is performed until the timing t1 + x, the reflector detection for the images (522, 523) acquired before the timing t1 + x may be delayed. The above delay may increase as the wearable device (101) continuously acquires images (521, 522, 523, 524) because the period for reflector detection is relatively long.

[0111] According to one embodiment, the wearable device (101) can identify areas (552, 553, 554) related to a reflector in other images (522, 523, 524) based on an area (551) in an image (521) acquired at a timing t1 + x and / or one or more feature points for distinguishing the area (551). The wearable device (101) can assign an identifier (ID) to each of the feature points in each of the images (521, 522, 523, 524). Feature points that are commonly included in at least two of the images (521, 522, 523, 524) can have the same identifier in the two images. For example, the f1 feature point common to all of the images (521, 522, 523, 524) may have an identifier that is matched across all of the images (521, 522, 523, 524) by the wearable device (101). For example, the f4 feature point common to all of the images (522, 523, 524) may have an identifier that is matched across all of the images (522, 523, 524). For example, the identifiers assigned to the feature points included in the image (521) may be different from the identifier assigned to the f4 feature point.

[0112] The wearable device (101) that has identified an area (551) for distinguishing feature points related to a reflector within an image (521) can use identifiers of feature points (e.g., f1 feature point) on the boundary line of the area (551) to confirm or identify an area corresponding to a reflector within another image. For example, by searching for feature points having identifiers of feature points (e.g., f1 feature point) on the boundary line of the area (551) within the image (522), the wearable device (101) can extract an area (552) within the image (522) for distinguishing feature points related to a reflector. Within the image (522), feature points f2 and f3, which were not extracted from the image (521), can be determined to be related to a reflector because they are included within the area (552).

[0113] For example, the wearable device (101) can search for feature points corresponding to feature points on the boundary line of the area (551) within the image (523) to identify an area (553) within the image (523) to distinguish feature points related to a reflector. Feature points f2, f3, f5, and f6 included in the area (553) can be determined to be related to the reflector.

[0114] Similarly, the wearable device (101) can identify a region (554) for distinguishing feature points associated with a reflector based on feature points within the image (524) that match feature points on the boundary of the region (551). For example, within the image (524), feature point f4 outside the region (554) can be determined as not associated with a reflector, and feature points f2 and f3 within the region (554) can be determined as associated with a reflector.

[0115] As described above, by comparing the feature points of sequentially acquired images (521, 522, 523, 524) in the time domain, the wearable device (101) can identify or determine areas (552, 553, 554) in other images (522, 523, 524) that match an area (551) related to a reflector in a specific image (521). For example, feature points for distinguishing areas (551, 552, 553, 554) (e.g., feature points positioned on the boundaries of areas (551, 552, 553, 554), such as the f1 feature point) can have matching identifiers in each of the images (521, 522, 523, 524). Based on the continuity of the identifiers of the feature points, the wearable device (101) can identify changes in the portion (512) related to the reflector within the images (521, 522, 523, 524).

[0116] The wearable device (101) can calculate the three-dimensional spatial coordinates of each of the feature points extracted from the images (521, 522, 523, 524). The wearable device (101) can calculate the three-dimensional spatial coordinates of each of the feature points based on whether each of the feature points is related to a reflector. For example, if the feature point is related to a reflector, the wearable device (101) can compensate for a reprojection error of the feature point due to the optical path of the reflected light. The embodiment is not limited thereto, and the wearable device (101) may preferentially calculate three-dimensional space coordinates of feature points different from feature points associated with a reflector among feature points in an image (e.g., images (521, 522, 523, 524)), or may bypass calculation of three-dimensional space coordinates for feature points associated with a reflector.

[0117] As described above, according to one embodiment, the wearable device (101) can track a portion (512) related to a reflector in each of the images (521, 522, 523, 524) using feature points in the images (521, 522, 523, 524) acquired from the camera. Based on the result of tracking the reflector, the wearable device (101) can calculate a position and / or direction of the wearable device (101) within the external space. Hereinafter, an operation of the wearable device (101) that calculates a position and / or direction based on feature points will be described with reference to FIGS. 6A, 6B, and / or 6C.

[0118] FIGS. 6A, 6B, and 6C illustrate an example of an operation of a wearable device (101) to identify a location of the wearable device (101) within an external space including a reflector. The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operations of the wearable device (101) described with reference to FIGS. 6A, 6B, and / or 6C.

[0119] Referring to FIGS. 6A, 6B, and / or 6C, different states (601, 602, 603) distinguished by the movement of a wearable device (101) and / or a user (110) wearing the wearable device (101) are illustrated. Referring to FIG. 6A, a state (601) of a user (110) entering a first external space (611) along a path (619) and then moving to a point (p1) outside the first external space (611) is illustrated. The first external space (611) and a second external space (612) next to the first external space (611) may be referred to as a room. Within the state (601), the wearable device (101) may identify fx feature points and / or fy feature points from images and / or videos acquired in the first external space (611). The diamond-shaped marks (◆) of FIG. 6a can represent three-dimensional positions of feature points extracted from images and / or videos for the first external space (611).

[0120] While worn by a user (110), the wearable device (101) may calculate, track, or monitor the position and / or orientation of the wearable device (101) based on the execution of the position tracker (271) (or head tracker) of FIG. 2A and / or FIG. 2B. The feature points indicated by the diamond-shaped marks (◆) of FIG. 6A may be used to generate map information for the first external space (611). The map information corresponding to the first external space (611) may include feature points and / or key frames of images and / or videos acquired within the first external space (611). The feature points included in the map information may be included in a point cloud for the first external space (611). A point cloud for the first external space (611) can be provided to a software application executed by the wearable device (101) based on a call to an API corresponding to the position tracker.

[0121] Referring to FIG. 6a, from images and / or videos acquired within the first external space (611), the wearable device (101) can determine that there is no reflector within the first external space (611).

[0122] Referring to FIG. 6B, a state (602) of a user (110) moving from a point (p1) along a path (629) to a point (p2) in a second external space (612) is illustrated. Based on images and / or videos acquired in the state (602), the wearable device (101) can identify feature points corresponding to landmarks in the second external space (612). The triangular marks (▲) and / or circular marks (○) in FIG. 6B can represent three-dimensional positions of feature points extracted from images and / or videos of the second external space (612).

[0123] Referring to FIG. 6B, from images and / or videos acquired within a second external space (612) including a reflector (621), the wearable device (101) can identify feature points associated with reflected light reflected by the reflector (621). Referring to FIG. 6B, as light for the f1 feature point, the f2 feature point, the f3 feature point, and the f4 feature point is reflected by the reflector (621), the wearable device (101) can identify the f1' feature point, the f2' feature point, the f3' feature point, and the f4' feature point corresponding to the f1 feature point, the f2' feature point, the f3' feature point, and the f4 feature point, respectively. Referring to FIG. 6B, the f1' feature point, the f2' feature point, the f3' feature point, and the f4' feature point based on the reflected light can be determined to be located beyond the reflector (621) with respect to the wearable device (101). According to one embodiment, the wearable device (101) may assign an attribute to feature points associated with reflected light (e.g., feature points f1', f2', f3', and f4' indicated by marks (○)) indicating that the feature points are associated with reflected light and / or a reflector (621). The wearable device (101) may store feature points acquired within the state (602) (e.g., feature points indicated by marks (▲, ○)) in map information associated with the second external space (612).

[0124] According to one embodiment, the wearable device (101) may compare feature points acquired within the state (602) (e.g., feature points indicated by marks (▲, ○)) with feature points identified based on the first external space (611) (e.g., feature points indicated by marks (◆)). For example, the wearable device (101) may compare the remaining feature points (e.g., feature points f1, f2, f3, f4, represented by marks (▲)) excluding feature points corresponding to reflected light of the reflector (621) (e.g., feature points f1', f2', f3', f4', represented by marks (○)) with feature points identified based on the first external space (611) (e.g., feature points indicated by marks (◆)). Based on the above comparison, the wearable device (101) can determine whether the user (110) has re-entered the first external space (611) (e.g., loop closing). Referring to FIG. 6B, since the feature points extracted from the image and / or video within the state (602) (e.g., feature points indicated by marks (▲)) are different from the feature points associated with the first external space (611) (e.g., feature points indicated by marks (◆)), the wearable device (101) can determine that the user (110) has not re-entered the first external space (611).

[0125] As described above, according to one embodiment, the wearable device (101) may use each of the feature points to calculate the position and / or orientation of the wearable device (101) based on the position of the feature points (e.g., the depth of the feature point relative to the wearable device (101)), the size of the reflector (621) identified from the image, and / or whether the feature point is associated with the reflector (621).

[0126] Referring to FIG. 6C, a state (603) of a user (110) moving from a point (p2) along a path (639) to a point (p3) within a first external space (611) is illustrated. Within the state (603), the wearable device (101) can identify the re-entry of the user (110) into the first external space (611) by searching for feature points (e.g., feature points indicated by marks (◆, ○)) having a three-dimensional position corresponding to the first external space (611) within an image and / or video. For example, the wearable device (101) can search for feature points represented by marks (◆, ○) in an image and / or video acquired from a camera (e.g., the camera (225) and / or the external camera (225-2) of FIG. 2A), excluding feature points related to a reflector (621) (e.g., feature points f1', f2', f3', f4' represented by marks (○)).

[0127] For example, if fx feature points and / or fy feature points are identified from images and / or videos acquired within the state (603), the wearable device (101) may determine that the user (110) has re-entered the first external space (611). If it is determined that the user (110) has re-entered the first external space (611), the wearable device (101) may more accurately calculate the position and / or orientation of the wearable device (101) within the first external space (611) based on the positions of the fx feature points and / or fy feature points identified within the images and / or videos. The wearable device (101) may compensate for an error in the position and / or orientation of the wearable device (101) that was previously calculated within the first external space (611) based on the calculated position and / or orientation. Compensation for the above error can be performed based on backpropagation.

[0128] For example, feature points f1', f2', f3', and f4' included in the first external space (611) may be excluded from the calculation of the position and / or orientation of the wearable device (101) (e.g., loop closing) because they have properties that appear to be related to the reflector (621). The embodiment is not limited thereto. In a first external space (611) different from a second external space (612) including the reflector (621), the wearable device (101) may exclude the feature points related to the reflector (621) from the calculation of the position and / or orientation of the wearable device (101).

[0129] Within a second external space (612) including a reflector (621), the wearable device (101) can calculate a position and / or orientation of the wearable device (101) using the feature points associated with the reflector (621). For example, feature points f1', f2', f3', and f4', which appear to be associated with the reflector (621), can be used to more accurately determine the position of the wearable device (101) relative to the reflector (621) within the second external space (612). For example, in order to reduce the error in the position and / or direction of the wearable device (101) calculated using the remaining feature points (e.g., feature points indicated by marks (▲)) excluding the feature points f1', f2', f3', and f4' associated with the reflector (621), the wearable device (101) may utilize the feature points f1', f2', f3', and f4' associated with the reflector (621).

[0130] In one embodiment, the wearable device (101) can perform a first loop closing within a state excluding feature points (e.g., feature points f1', f2', f3', f4' indicated by marks (○)) associated with the reflector (621) and having three-dimensional positions within the first external space (611). Based on the first loop closing, the wearable device (101) can search for feature points indicated by marks (◆, ○) among the feature points indicated by marks (◆) within images and / or videos acquired from the camera. Based on the search, the wearable device (101) can determine whether the user (110) has re-entered the first external space (611).

[0131] The wearable device (101) that performed the first loop closing can perform the second loop closing based on the feature points associated with the reflector (621) and having three-dimensional positions within the first external space (611). Based on the second loop closing, the wearable device (101) can search for the feature points indicated by marks (◆, ○) within the images and / or videos acquired from the camera. Based on the search, the wearable device (101) can determine whether the user (110) has re-entered the first external space (611). Based on the first loop closing and / or the second loop closing, the wearable device (101) can more accurately determine the location of the user (110). For example, if the location of the user (110) is not determined based on the second loop closing, the wearable device (101) may determine the location of the user (110) based on the first loop closing. The failure to determine the location of the user (110) based on the loop closing may include cases where less than a certain number and / or a certain percentage of feature points are retrieved from the image and / or video.

[0132] In one embodiment, the wearable device (101) may use feature points (e.g., feature points f1', f2', f3', and f4' indicated by marks (○)) associated with the reflector (621) to calculate the position and / or orientation of the wearable device (101) based on the priority of the feature points. The priority may be related to the number of times the feature points are extracted from images acquired at different timings. For example, the greater the number of times, the higher the priority the feature point may have. The wearable device (101) may use feature points having relatively high priorities to calculate the position and / or orientation of the wearable device (101).

[0133] In one embodiment, the wearable device (101) may use feature points associated with the reflector (621) to calculate the position and / or orientation of the wearable device (101) based on the shape and / or reflective properties of the reflector (621). For example, if the reflective properties of the reflector (621) are not uniform and / or the reflector (621) has a curved shape, the wearable device (101) may not use feature points associated with the reflector (621) to calculate the position and / or orientation of the wearable device (101). For example, the non-uniform reflective properties and / or curved shape of the reflector (621) may cause distortion of a portion corresponding to the reflector (621) in an image acquired by the wearable device (101). The wearable device (101) may discard, ignore, or bypass computations based on the feature points extracted from the distorted portion so that the feature points are not used to calculate the position and / or orientation of the wearable device (101). The shape and / or reflection characteristics of the reflector (621) may be included in the reflector information (420) acquired based on the neural network (410) of FIG. 4.

[0134] In one embodiment, the wearable device (101) may use the feature point relative to the reflector (621) to calculate the position and / or orientation of the wearable device (101). For example, if the depth of the feature point (or the distance of the feature point relative to the reflector (621)) exceeds a threshold distance, the wearable device (101) may use other feature points (e.g., feature points not associated with the reflector (621)) excluding the feature point to calculate the position and / or orientation of the wearable device (101), and may not use the feature point to calculate the position and / or orientation of the wearable device (101).

[0135] As described above, according to one embodiment, the wearable device (101) can identify the position and / or orientation of the wearable device (101) using images and / or videos acquired at a specific point in time. The wearable device (101) can acquire images and / or videos from a camera to calculate the position and / or orientation of the wearable device (101) independently of the error of a sensor (e.g., a GPS sensor and / or an IMU) that gradually increases in the time domain. The wearable device (101) can identify at least a portion associated with a reflector (621) within the images and / or videos. Using the at least a portion, the wearable device (101) can reduce the error in the position and / or orientation of the wearable device (101) caused by the reflector (621).

[0136] As described above, according to one embodiment, the wearable device (101) can identify an external space including the wearable device (101) among the first external space (611) and / or the second external space (612), based on feature points of an image acquired using a camera. The wearable device (101) can obtain information related to the first external space (611) (e.g., the position and / or direction of the wearable device (101) within the first external space (611)) based on feature points (e.g., feature points indicated by marks (◆, ○)) having coordinate values ​​corresponding to the first external space (611) and different feature points (e.g., feature points indicated by marks (◆)) from the feature points related to the reflector (621) based on the other feature points.

[0137] The wearable device (101) can obtain information related to the second external space (612) (e.g., the position and / or direction of the wearable device (101) within the second external space (612)) by using feature points (e.g., feature points indicated by marks (▲)) having coordinate values ​​corresponding to the second external space (612) based on identifying the position of the wearable device (101) included in the second external space (612). While positioned within the second external space (612) including the reflector (621), the wearable device (101) can calculate or identify the position and / or direction of the wearable device (101) within the second external space (612) by using feature points (e.g., feature points indicated by marks (○)) associated with the reflector (621).

[0138] Below, with reference to FIG. 7, a UI displayed by a wearable device (101) that identifies a reflector (621) is exemplarily described.

[0139] Fig. 7 illustrates an example of a user interface (UI) displayed by a wearable device (101) that has identified a reflector (130). The wearable device (101) of Figs. 1, 2a, and 2b and / or the processor (210) of Fig. 2a can perform the operation of the wearable device (101) described with reference to Fig. 7.

[0140] According to one embodiment, the wearable device (101) may display a screen (710) including images and / or videos acquired from a camera (e.g., camera (225) and / or external camera (225-2) of FIG. 2A). The screen (710) may be displayed on the display (e.g., display (220) of FIG. 2A) of the wearable device (101) based on the execution of a software application based on AR, MR, and / or VST. Within the screen (710), the wearable device (101) may display a virtual object (720) together with visual objects (141, 142, 143, 144) corresponding to external objects. The virtual object (720) may have a three-dimensional position mapped to an external space viewable through the screen (710). For example, the wearable device (101) may provide a visual effect of floating within external space based on movement and / or rotation of a virtual object (720) within the screen (710).

[0141] The wearable device (101) can identify a reflector (130) adjacent to the wearable device (101) based on an image and / or video acquired from a camera. The wearable device (101) that has identified a portion related to the reflector (130) within the image can display a visual object on the display of the wearable device (101) for verifying the reflector (130). In the exemplary screen (710) of FIG. 7, the wearable device (101) can highlight a portion (732) corresponding to the reflector (130) using a visual object in the form of a line and / or shape. The wearable device (101) can display a virtual object (730) for notifying the identification of the reflector (130). The virtual object (730) may include designated text indicating the identification of the reflector (130) (e.g., “Reflector Detected”).

[0142] In one embodiment, the wearable device (101) may determine whether to display a virtual object (730) based on the number of feature points in a first portion corresponding to the reflector (130) and a second portion (e.g., a portion adjacent to the first portion) that are different from the first portion in the image, if the wearable device (101) recognizes or identifies a reflector (130) in the image. For example, if feature points exceeding a specified number are extracted in the second portion, the wearable device (101) may not display the virtual object (730). In the example above, if feature points less than a specified number are extracted in the second portion, the wearable device (101) may display the virtual object (730). For example, if feature points less than a specified number are extracted in the second portion during a specified period, the wearable device (101) may display the virtual object (730). In the above example, if the number of feature points extracted exceeds the specified number in the second part, the wearable device (101) may not display the virtual object (730).

[0143] The wearable device (101) that has identified the reflector (130) may perform an operation to additionally acquire information related to the reflector (130). For example, the wearable device (101) may guide the acquisition of images and / or videos of an external space adjacent to the reflector (130) based on the virtual object (730). For example, the wearable device (101) may guide the user (110) wearing the wearable device (101) to move to a location adjacent to the reflector (130). Based on the above operation, the wearable device (101) may additionally acquire feature points related to the reflector (130) and / or the external space adjacent to the reflector (130). The feature points additionally acquired by the wearable device (101) may be used to more accurately generate map information about the external space including the reflector (130).

[0144] Within an exemplary state in which a reflector (130) is identified, the wearable device (101) can provide a user experience based on the reflector (130). For example, the wearable device (101) can provide a function associated with a virtual object (720) using a visual object (143) and / or a portion (732) within the screen (710) corresponding to the reflector (130). For example, the wearable device (101) can provide a visual effect such as a virtual object (720) being attached to the visual object (143) based on a distance between the visual object (143) and the virtual object (720). While the virtual object (720) and the visual object (143) are connected to each other, the wearable device (101) can move the virtual object (720) on the visual object (143) based on a virtual friction coefficient associated with the reflector (130). For example, on a visual object (143), a virtual object (720) may slide at a relatively fast speed. The embodiment is not limited thereto, and the wearable device (101) may execute a function of replicating a virtual object (720) based on contact between the visual object (143) and the virtual object (720).

[0145] For example, the wearable device (101) can display a mirror image corresponding to a virtual object (720) on a portion (732) within a screen (710) associated with a reflector (130). Based on the mirror image displayed on the portion (732), the wearable device (101) can provide a visual effect such that the virtual object (720) is reflected on the reflector (130).

[0146] As described above, according to one embodiment, the wearable device (101) can identify a reflector (130) in an external space. Based on the result of identifying the reflector (130), the wearable device (101) can prevent the degradation of the position tracking (e.g., head tracking) performance due to the reflector (130). For example, the wearable device (101) can improve the position tracking performance by using feature points related to the reflected light of the reflector (130). Considering the amount of computation required to identify a portion (732) corresponding to the reflector (130) (e.g., the amount of computation for driving the neural network (410) of FIG. 4), the wearable device (101) can use a portion (732) acquired from an image of a specific timing to identify a portion (732) corresponding to the reflector (130) from an image of a different timing.

[0147] Hereinafter, with reference to FIG. 8, an exemplary operation of a wearable device (101) according to one embodiment is described.

[0148] FIG. 8 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIG. 2A may perform the operations described with reference to FIG. 8. At least one of the operations of FIG. 8 may be related to an operation of a wearable device executing a location tracker (271) of FIG. 2A and / or FIG. 2B. The operation of FIG. 8 may be related to the operation of FIG. 3.

[0149] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0150] Referring to FIG. 8, within operation (810), a processor of a wearable device according to one embodiment may obtain an image of an external space (e.g., images (140, 150) of FIG. 1, images (521, 522, 523, 524) of FIG. 5a) using a camera (e.g., camera (225) of FIG. 2a and / or external camera (225-2)).

[0151] Referring to FIG. 8, within operation (820), the processor of the wearable device according to one embodiment can identify whether at least a portion of the image of operation (810) is associated with a reflector that reflects light (e.g., reflector (130) of FIG. 1, reflector (510) of FIG. 5A, and / or reflector (621) of FIGS. 6A-6C). The processor can identify a portion of the image associated with the reflector (e.g., portion (430) of FIG. 4, portion (512) of FIGS. 5A-5B, and / or portion (732) of FIG. 7) based on the execution of the reflector detector (295) of FIG. 2B and / or FIG. 4. Within the state of having identified at least a portion of the image associated with the reflector of operation (820-Yes), the processor of the wearable device can perform operation (830). If at least a portion of the image associated with the reflector of operation (820) is not identified (820-No), the processor of the wearable device may perform operation (840).

[0152] Referring to FIG. 8, in operation (830), a processor of a wearable device according to an embodiment may obtain information indicating whether each of the feature points of an image is included in at least a portion of the image associated with a reflector. The information of operation (830) may include attributes of each of the feature points. The information may include a logical value (e.g., a Boolean data type), a flag, and / or a probability value indicating whether the feature point corresponds to a reflector.

[0153] Referring to FIG. 8, in operation (840), according to one embodiment, a processor of a wearable device may recognize an external space based on feature points of an image. The processor may identify a shape of the external space and / or one or more external objects included in the external space based on three-dimensional positions of the feature points. Based on operation (840), the processor may generate or obtain information representing the external space. The result of recognizing the external space may include map information (and / or a local map) corresponding to the external space.

[0154] Referring to FIG. 8, in operation (850), a processor of a wearable device according to an embodiment may provide a virtual space at least partially mapped to an external space based on a result of recognizing an external space. The processor may execute instructions for providing the virtual space, which are included in a software application installed on the wearable device (e.g., a software application included in the application layer (240) of FIG. 2A). By executing the instructions based on a result of recognizing an external space in operation (840), the processor may obtain a virtual space at least partially mapped to the external space. The processor may display at least a portion of the obtained virtual space on a display (e.g., the display (220) of FIG. 2A).

[0155] Hereinafter, exemplary appearances of wearable devices described with reference to FIGS. 1 to 8 are illustrated with reference to FIGS. 9A, 9B, 10A, and / or 10B. The wearable device (900) of FIG. 9A and / or 9B and / or the wearable device (1000) of FIG. 10A and / or 10B may be an example of the wearable device (101) of FIG. 1.

[0156] FIG. 9A illustrates an example of a perspective view of a wearable device, according to one embodiment. According to one embodiment, the wearable device (900) may have the form of glasses that are wearable on a body part of a user (e.g., the head). The wearable device (900) may include a head-mounted display (HMD). For example, the housing of the wearable device (900) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (900) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0157] Referring to FIG. 9A, according to one embodiment, a wearable device (900) may include at least one display (950) and a frame (909) supporting at least one display (950).

[0158] According to one embodiment, a wearable device (900) can be worn on a part of a user's body. The wearable device (900) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (900). For example, the wearable device (900) can display a virtual reality image provided from at least one optical device (982, 984) of FIG. 9B on at least one display (950) in response to a user's designated gesture acquired through the motion recognition cameras (960-2, 960-3) of FIG. 9B.

[0159] According to one embodiment, at least one display (950) may provide visual information to a user. For example, at least one display (950) may include a transparent or translucent lens. At least one display (950) may include a first display (950-1) and / or a second display (950-2) spaced apart from the first display (950-1). For example, the first display (950-1) and the second display (950-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0160] Referring to FIG. 9B, at least one display (950) can provide visual information transmitted from external light to a user through a lens included in the at least one display (950), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (950) can include a first surface (931) and a second surface (932) opposite to the first surface (931). A display area can be formed on the second surface (932) of the at least one display (950). When a user wears the wearable device (900), external light can be transmitted to the user by being incident on the first surface (931) and transmitted through the second surface (932). As another example, at least one display (950) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (982, 984) on a real screen transmitted through external light, in a display area formed on the second surface (932).

[0161] In one embodiment, at least one display (950) may include at least one waveguide (933, 934) that diffracts light emitted from at least one optical device (982, 984) and transmits the diffracted light to a user. The at least one waveguide (933, 934) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (933, 934). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (933, 934) may be propagated to the other end of the at least one waveguide (933, 934) by the nano-pattern. At least one waveguide (933, 934) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (933, 934) may be arranged within the wearable device (900) to guide a screen displayed by at least one display (950) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (933, 934).

[0162] The wearable device (900) can analyze an object included in a real image collected through a shooting camera (960-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (950). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (900) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (900) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using the multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (900) can view an image displayed on at least one display (950).

[0163] According to one embodiment, the frame (909) may be formed as a physical structure that allows the wearable device (900) to be worn on the user's body. According to one embodiment, the frame (909) may be configured so that, when the user wears the wearable device (900), the first display (950-1) and the second display (950-2) can be positioned corresponding to the user's left and right eyes. The frame (909) may support at least one display (950). For example, the frame (909) may support the first display (950-1) and the second display (950-2) to be positioned corresponding to the user's left and right eyes.

[0164] Referring to FIG. 9A, the frame (909) may include a region (920) that at least partially contacts a portion of the user's body when the user wears the wearable device (900). For example, the region (920) of the frame (909) that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (900) makes contact with. According to one embodiment, the frame (909) may include a nose pad (910) that contacts a portion of the user's body. When the wearable device (900) is worn by the user, the nose pad (910) may contact a portion of the user's nose. The frame (909) may include a first temple (904) and a second temple (905) that contact a portion of the user's body that is distinct from the portion of the user's body.

[0165] For example, the frame (909) may include a first rim (901) that surrounds at least a portion of the first display (950-1), a second rim (902) that surrounds at least a portion of the second display (950-2), a bridge (903) that is disposed between the first rim (901) and the second rim (902), a first pad (911) that is disposed along a portion of the edge of the first rim (901) from one end of the bridge (903), a second pad (912) that is disposed along a portion of the edge of the second rim (902) from the other end of the bridge (903), a first temple (904) that extends from the first rim (901) and is secured to a portion of an ear of the wearer, and a second temple (905) that extends from the second rim (902) and is secured to a portion of an ear opposite the ear. The first pad (911) and the second pad (912) may be in contact with a portion of the user's nose, and the first temple (904) and the second temple (905) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (904, 905) may be rotatably connected to the rim through the hinge units (906, 907) of FIG. 9B. The first temple (904) may be rotatably connected to the first rim (901) through the first hinge unit (906) disposed between the first rim (901) and the first temple (904). The second temple (905) may be rotatably connected to the second rim (902) through the second hinge unit (907) disposed between the second rim (902) and the second temple (905). According to one embodiment, the wearable device (900) can identify an external object (e.g., a user's fingertip) touching the frame (909) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (909).

[0166] According to one embodiment, the wearable device (900) may include hardwares that perform various functions (e.g., hardwares described above based on the block diagrams of FIGS. 2A, 2B, and / or 4). For example, the hardwares may include a battery module (970), an antenna module (975), at least one optical device (982, 984), speakers (e.g., speakers 955-1, 955-2), microphones (e.g., microphones 965-1, 965-2, 965-3), a light-emitting module, and / or a printed circuit board (PCB) (990) (e.g., a printed circuit board). The various hardwares may be disposed within the frame (909).

[0167] According to one embodiment, the microphones (e.g., microphones 965-1, 965-2, 965-3) of the wearable device (900) may be disposed on at least a portion of the frame (909) to acquire sound signals. A first microphone (965-1) disposed on the bridge (903), a second microphone (965-2) disposed on the second rim (902), and a third microphone (965-3) disposed on the first rim (901) are illustrated in FIG. 9B , but the number and arrangement of the microphones (965) are not limited to the embodiment of FIG. 9B . When the number of microphones (965) included in the wearable device (900) is two or more, the wearable device (900) may use multiple microphones disposed on different portions of the frame (909) to identify the direction of the sound signal.

[0168] In one embodiment, at least one optical device (982, 984) can project a virtual object onto at least one display (950) to provide various image information to a user. For example, at least one optical device (982, 984) can be a projector. At least one optical device (982, 984) can be positioned adjacent to at least one display (950) or can be included within at least one display (950) as a part of at least one display (950). In one embodiment, the wearable device (900) can include a first optical device (982) corresponding to a first display (950-1) and a second optical device (984) corresponding to a second display (950-2). For example, at least one optical device (982, 984) may include a first optical device (982) disposed at an edge of a first display (950-1) and a second optical device (984) disposed at an edge of a second display (950-2). The first optical device (982) may transmit light to a first waveguide (933) disposed on the first display (950-1), and the second optical device (984) may transmit light to a second waveguide (934) disposed on the second display (950-2).

[0169] In one embodiment, the camera (960) may include a recording camera (960-4), an eye tracking camera (ET CAM) (960-1), and / or a motion recognition camera (960-2, 960-3). The recording camera (960-4), the eye tracking camera (960-1), and the motion recognition cameras (960-2, 960-3) may be positioned at different locations on the frame (909) and may perform different functions. The eye tracking camera (960-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (900). For example, the wearable device (900) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (960-1).

[0170] The wearable device (900) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (960-1). The wearable device (900) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (900) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (960-1). The wearable device (900) can render an image (or screen) displayed on at least one display (950) based on the position of the user's eyes.

[0171] For example, the visual quality of a first region related to the gaze within an image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinct from the first region may be different from each other. The wearable device (900) may obtain an image having the visual quality of the first region matching the user's gaze and the visual quality of the second region using foveated rendering. For example, if the wearable device (900) supports an iris recognition function, user authentication may be performed based on iris information obtained using the gaze tracking camera (960-1). Although an example in which the gaze tracking camera (960-1) is positioned toward the user's right eye is illustrated in FIG. 9B, the embodiment is not limited thereto, and the gaze tracking camera (960-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0172] In one embodiment, the capturing camera (960-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (960-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (960-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (950). The at least one display (950) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (960-4) is superimposed on a virtual image provided through at least one optical device (982, 984). The wearable device (900) can compensate for depth information (e.g., the distance between the wearable device (900) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (960-4). The wearable device (900) can perform object recognition through an image acquired using the capturing camera (960-4). The wearable device (900) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capturing camera (960-4). The wearable device (900) can perform a pass-through function to display an image acquired through the capturing camera (960-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (950). In one embodiment, the capturing camera (960-4) can be disposed on a bridge (903) disposed between the first rim (901) and the second rim (902).

[0173] The gaze tracking camera (960-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (900) and matching the user's gaze with visual information provided to at least one display (950). For example, when the wearable device (900) looks straight ahead, the wearable device (900) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (950). The gaze tracking camera (960-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (960-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (960-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (960-1) may be positioned within the first rim (901) and / or the second rim (902) to face the direction in which the user wearing the wearable device (900) is positioned.

[0174] The gesture recognition camera (960-2, 960-3) can recognize the movement of the user's entire body or a part of the user's body, such as the user's torso, hand, or face, and thereby provide a specific event on a screen provided on at least one display (950). The gesture recognition camera (960-2, 960-3) can recognize the user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (950). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (960-2, 960-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor can perform a gesture recognition function and / or an object tracking function using the gesture recognition camera (960-2, 960-3). In one embodiment, the motion recognition cameras (960-2, 960-3) may be positioned on the first rim (901) and / or the second rim (902).

[0175] The camera (960) included in the wearable device (900) is not limited to the above-described gaze tracking camera (960-1) and motion recognition cameras (960-2, 960-3). For example, the wearable device (900) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (900) can identify an external object based on a sensor for identifying the distance between the wearable device (900) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (960) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (900) may include a camera (960) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (900).

[0176] Although not shown, in one embodiment, the wearable device (900) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (960). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (909) and the hinge units (906, 907).

[0177] In one embodiment, the battery module (970) may supply power to the electronic components of the wearable device (900). In one embodiment, the battery module (970) may be disposed within the first temple (904) and / or the second temple (905). For example, the battery module (970) may be a plurality of battery modules (970). The plurality of battery modules (970) may be disposed within each of the first temple (904) and the second temple (905). In one embodiment, the battery module (970) may be disposed at an end of the first temple (904) and / or the second temple (905).

[0178] The antenna module (975) can transmit signals or power to the outside of the wearable device (900), or receive signals or power from the outside. In one embodiment, the antenna module (975) can be positioned within the first temple (904) and / or the second temple (905). For example, the antenna module (975) can be positioned close to one surface of the first temple (904) and / or the second temple (905).

[0179] The speaker (955) can output an audio signal to the outside of the wearable device (900). The audio output module may be referred to as a speaker. In one embodiment, the speaker (955) may be positioned within the first temple (904) and / or the second temple (905) so as to be positioned adjacent to the ear of a user wearing the wearable device (900). For example, the speaker (955) may include a second speaker (955-2) positioned within the first temple (904) and thus adjacent to the user's left ear, and a first speaker (955-1) positioned within the second temple (905) and thus adjacent to the user's right ear.

[0180] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (900) to the user. For example, when the wearable device (900) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (901) and / or the second rim (902).

[0181] Referring to FIG. 9B, according to one embodiment, a wearable device (900) may include a printed circuit board (PCB) (990). The PCB (990) may be included in at least one of the first temple (904) or the second temple (905). The PCB (990) may include an interposer positioned between at least two sub-PCBs. One or more hardwares included in the wearable device (900) (e.g., hardwares illustrated by different blocks in FIG. 2A) may be positioned on the PCB (990). The wearable device (900) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0182] According to one embodiment, a wearable device (900) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (900) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (900). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (900) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (900) based on the IMU.

[0183] FIGS. 10A and 10B illustrate an example of an exterior appearance of a wearable device according to one embodiment. The wearable device (1000) of FIGS. 10A and 10B may include at least a portion of the hardware of the wearable device (900) described with reference to FIGS. 9A and / or 9B . An example of an exterior appearance of a first side (1010) of a housing of the wearable device (1000) according to one embodiment is illustrated in FIG. 10A , and an example of an exterior appearance of a second side (1020) opposite to the first side (1010) may be illustrated in FIG. 10B .

[0184] Referring to FIG. 10A, a first surface (1010) of a wearable device (1000) according to one embodiment may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1000) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (904) and / or the second temple (905) of FIGS. 9A and 9B). A first display (950-1) for outputting an image to a left eye among the user's two eyes, and a second display (950-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (1010). The wearable device (1000) is formed on the first surface (1010) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (950-1) and the second display (950-2).

[0185] According to one embodiment, a wearable device (1000) may include cameras (960-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (950-1) and the second display (950-2). The cameras (960-1) may be referred to as the gaze tracking camera (960-1) of FIG. 9B. According to one embodiment, a wearable device (1000) may include cameras (960-5, 960-6) for photographing and / or recognizing a face of a user. The cameras (960-5, 960-6) may be referred to as FT cameras. The wearable device (1000) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (960-5, 960-6). For example, the wearable device (1000) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar representing a human face) using information obtained by cameras (960-5, 960-6) (e.g., FT cameras) and representing a facial expression of a user wearing the wearable device (1000).

[0186] Referring to FIG. 10B, a camera (e.g., cameras (960-7, 960-8, 960-9, 960-10, 960-11, 960-12)) and / or a sensor (e.g., a depth sensor (1030)) for obtaining information related to the external environment of the wearable device (1000) may be disposed on a second surface (1020) opposite to the first surface (1010) of FIG. 10A. For example, the cameras (960-7, 960-8, 960-9, 960-10) may be disposed on the second surface (1020) for recognizing external objects. The cameras (960-7, 960-8, 960-9, 960-10) of FIG. 10b can correspond to the motion recognition cameras (960-2, 960-3) of FIG. 9b.

[0187] For example, using cameras (960-11, 960-12), the wearable device (1000) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (960-11) can be placed on the second face (1020) of the wearable device (1000) to obtain an image to be displayed through the second display (950-2) corresponding to the right eye among the two eyes. The camera (960-12) can be placed on the second face (1020) of the wearable device (1000) to obtain an image to be displayed through the first display (950-1) corresponding to the left eye among the two eyes. The cameras (960-11, 960-12) can correspond to the shooting camera (960-4) of FIG. 9B.

[0188] According to one embodiment, the wearable device (1000) may include a depth sensor (1030) disposed on the second face (1020) to identify a distance between the wearable device (1000) and an external object. Using the depth sensor (1030), the wearable device (1000) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (1000). Although not illustrated, a microphone may be disposed on the second face (1020) of the wearable device (1000) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0189] In one embodiment, a method for identifying a position and / or orientation of a wearable device within an external space including a reflector may be required. As described above, a wearable device according to one embodiment (e.g., wearable device (101) of FIG. 1 , wearable device (900) of FIGS. 9A and 9B , and / or wearable device (1000) of FIGS. 10A and 10B ) may include a camera (e.g., camera (225) of FIG. 2A ), a memory for storing instructions including one or more storage media (e.g., memory (215) of FIG. 2A ), and at least one processor (e.g., processor (210) of FIG. 2A ). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to acquire, using the camera, an image of an external space including the wearable device (e.g., images (140, 150) of FIG. 1, images (521, 522, 523, 524) of FIG. 5A). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a portion of the image (e.g., a portion (430) of FIG. 4 , a portion (512) of FIGS. 5A - 5B , and / or a portion (732) of FIG. 7 ) associated with a reflector (e.g., a reflector (130) of FIG. 1 , a reflector (510) of FIG. 5A , and / or a reflector (621) of FIGS. 6A - 6C ) that reflects light based on object recognition of the image. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to execute a software application using information based on whether each of the feature points of the image is included in the portion to provide a virtual space at least partially mapped to the external space.

[0190] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine, based on a size of the portion within the image, whether at least one feature point identified from the portion among the feature points is used to identify a location of the wearable device within the external space.

[0191] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device, in response to the portion associated with the reflector, to display a visual object on a display of the wearable device for verifying the reflector.

[0192] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, among the feature points of the image acquired at a first timing, first feature points surrounding the portion. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, within another image acquired at a second timing different from the first timing, another part associated with the reflector based on second feature points that are included in the other image and that match the first feature points.

[0193] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify the second feature points that match the first feature points based on identifiers assigned to each of the feature points included in the other image.

[0194] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to execute instructions of the software application for reproducing the virtual space based on the information, thereby displaying at least a portion of the virtual space linked to the external space on the display of the wearable device.

[0195] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a location of the wearable device within the virtual space based on a location of the wearable device within the external space indicated by the information.

[0196] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a location of the portion within the image using a neural network into which the image has been input.

[0197] According to one embodiment, a method of a wearable device as described above may include an operation of obtaining an image of an external space including the wearable device using a camera of the wearable device (e.g., operation (810) of FIG. 8). The method may include an operation of identifying a portion of the image associated with a reflector that reflects light based on object recognition of the image. The method may include an operation of executing a software application using information based on whether each of the feature points of the image is included in the portion to provide a virtual space at least partially mapped to the external space (e.g., operation (850) of FIG. 8).

[0198] For example, the method may include an operation of determining, based on a size of the portion within the image, whether at least one feature point identified from the portion, among the feature points, is used to identify a location of the wearable device within the external space. The method may include an operation of obtaining the information representing the external space based on whether the at least one feature point is used to identify the location.

[0199] For example, the identifying action may include, in response to the portion associated with the reflector, displaying a visual object on the display of the wearable device for identifying the reflector.

[0200] For example, the identifying operation may include an operation of identifying first feature points surrounding the portion among the feature points of the image acquired at a first timing. The method may include an operation of identifying another portion related to the reflector within the other image based on second feature points included in another image acquired at a second timing different from the first timing and matching the first feature points.

[0201] For example, the act of identifying the other part may include an act of identifying the second feature points that match the first feature points based on identifiers assigned to each of the feature points included in the other image.

[0202] For example, the provided operation may include an operation of executing instructions of the software application for reproducing the virtual space based on the information, and displaying at least a portion of the virtual space linked to the external space on the display of the wearable device.

[0203] For example, the provided action may include an action of determining a location of the wearable device within the virtual space based on a location of the wearable device within the external space indicated by the information.

[0204] For example, the identifying action may include an action of identifying the location of the portion within the image using a neural network into which the image is input.

[0205] As described above, according to one embodiment, a wearable device (e.g., wearable device (101) of FIG. 1, wearable device (900) of FIGS. 9A and 9B, and / or wearable device (1000) of FIGS. 10A and 10B) may include a camera (e.g., camera (225) of FIG. 2A), a memory for storing instructions (e.g., memory (215) of FIG. 2A), and a processor for executing the instructions (e.g., processor (210) of FIG. 2A). The processor may be configured to identify an external space including the wearable device based on first feature points of an image acquired using the camera (e.g., images (140, 150) of FIG. 1, images (521, 522, 523, 524) of FIG. 5A). The processor may be configured to identify second feature points, which have coordinate values ​​corresponding to the first external space and have been identified based on the camera, based on identifying the wearable device included in the first external space next to the second external space, among a first external space (e.g., the first external space (611) of FIGS. 6A to 6C) or a second external space (e.g., the second external space (612) of FIGS. 6A to 6C) including a reflector (e.g., the reflector (130) of FIG. 1, the reflector (510) of FIG. 5A, and / or the reflector (621) of FIGS. 6A to 6C). The processor may be configured to obtain information related to the first external space based on the first feature points and fourth feature points, which are different from one or more third feature points associated with the reflector, among the second feature points.The processor may be configured to obtain information related to the second external space using the one or more third characteristic points and the first characteristic points based on identifying the wearable device included in the second external space among the first external space or the second external space.

[0206] For example, the processor may be configured to display, on the display of the wearable device, at least a portion of a virtual space mapped to an external space including the wearable device among the first external space or the second external space, based on the acquired information.

[0207] For example, the processor may be configured to identify the first feature points within the image using a neural network into which the image is input.

[0208] For example, the processor may be configured to identify one or more third feature points among the second feature points based on attributes of the second feature points.

[0209] According to one embodiment of the present invention, a method of a wearable device as described above may include an operation of identifying an external space including a wearable device based on first feature points of an image acquired using a camera of the wearable device. The method may include an operation of identifying second feature points, which have been identified based on the camera and have coordinate values ​​corresponding to the first external space, based on identifying the wearable device included in the first external space next to the second external space, among the first external space or a second external space including a reflector. The method may include an operation of obtaining information associated with the first external space based on the first feature points and fourth feature points, which are different from one or more third feature points associated with the reflector, among the second feature points. The method may include an operation of obtaining information related to the second external space using the one or more third characteristic points and the first characteristic points based on identifying the wearable device included in the second external space among the first external space or the second external space.

[0210] For example, the method may include an operation of displaying, on a display of the wearable device, at least a portion of a virtual space mapped to an external space including the wearable device among the first external space or the second external space, based on the acquired information.

[0211] For example, the operation of identifying the external space may include an operation of identifying the first feature points within the image using a neural network into which the image is input.

[0212] For example, the operation of obtaining the information related to the first external space may include an operation of identifying one or more third feature points among the second feature points based on attributes of the second feature points.

[0213] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0214] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0215] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0216] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0217] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In a wearable device (101; 900; 1000), Camera (225); A memory (215) for storing instructions, including one or more storage media; and At least one processor (210) comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Using the above camera, an image (140, 150; 521, 522, 523, 524) of an external space including the wearable device is acquired; Based on object recognition for the image, identifying a portion (430; 512; 732) of the image associated with a reflector (130; 510; 621) that reflects light; and Causing a software application to execute information based on whether each of the feature points of the image is included in the part, thereby providing a virtual space at least partially mapped to the external space. Wearable devices.

2. In claim 1, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on the size of the portion within the image, determining whether at least one feature point identified from the portion among the feature points is used to identify a location of the wearable device within the external space. Wearable devices.

3. In claims 1 and 2, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: In response to said portion associated with said reflector, causing a visual object to be displayed on a display of said wearable device for verifying said reflector; Wearable devices.

4. In claims 1 to 3, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Among the feature points of the image acquired at the first timing, first feature points surrounding the portion are identified; causing another part related to the reflector to be identified within the other image based on second feature points, which are included in another image acquired at a second timing different from the first timing and which match the first feature points; Wearable devices.

5. In claims 1 to 4, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing to identify the second feature points matching the first feature points based on the identifiers assigned to each of the feature points included in the other image. Wearable devices.

6. In claims 1 to 5, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing instructions of the software application for reproducing the virtual space to be executed based on the information, thereby causing at least a part of the virtual space linked with the external space to be displayed on the display of the wearable device. Wearable devices.

7. In claims 1 to 6, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing to determine the location of the wearable device within the virtual space based on the location of the wearable device within the external space indicated by the information. Wearable devices.

8. In claims 1 to 7, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Using a neural network into which the above image is input, the location of the above part within the image is identified, Wearable devices.

9. In the method of a wearable device, An operation (810) of obtaining an image of an external space including the wearable device using a camera of the wearable device; An operation of identifying a portion of the image associated with a reflector that reflects light, based on object recognition of the image; and An operation (850) comprising executing a software application using information based on whether each of the feature points of the image is included in the part, thereby providing a virtual space at least partially mapped to the external space. method.

10. In claim 9, An operation of determining, based on a size of the portion within the image, whether at least one feature point identified from the portion among the feature points is used to identify a location of the wearable device within the external space; and Further comprising an operation of obtaining the information representing the external space based on whether the at least one feature point is used to identify the location. method.

11. In claims 9 to 10, the identifying action comprises: In response to the above part associated with the reflector, comprising an action of displaying a visual object for identifying the reflector on a display of the wearable device; method.

12. In claims 9 to 11, the identifying action comprises: An operation of identifying first feature points surrounding the portion among the feature points of the image acquired at the first timing; An operation of identifying another part related to the reflector within the other image based on second feature points, which are included in another image acquired at a second timing different from the first timing and which match the first feature points, method.

13. In claims 9 to 12, the operation of identifying the other part comprises: An operation of identifying the second feature points matching the first feature points based on identifiers assigned to each of the feature points included in the other image, method.

14. In claims 9 to 13, the operation provided is: An operation of displaying at least a portion of the virtual space linked with the external space on the display of the wearable device by executing instructions of the software application for reproducing the virtual space based on the information, method.

15. In claims 9 to 14, the provided operation comprises: An operation of determining a position of the wearable device within the virtual space based on a position of the wearable device within the external space indicated by the information, method.

Images