Wearable device for acquiring information on external space including reflector and method thereof

Abstract

A wearable device according to an embodiment may include: a camera; a memory to store instructions; and a processor to execute the instructions. The processor may be configured to obtain an image of an external space including the wearable device by using the camera. The processor may be configured to identify a portion of the image associated with a reflector that reflects light based on identifying an object in the image. The processor may be configured to execute the software application using information based on whether each feature point in the image is included in the portion to provide a virtual space that is at least partially mapped to the external space.

Description

Technical Field

[0001] This disclosure relates to a wearable device and method for obtaining information about an external space including a reflector. Background Technology

[0002] To provide an enhanced user experience (UX), an electronic device is being developed that delivers augmented reality (AR) services, displaying information generated by a computer in conjunction with external objects in the real world. The electronic device can be a wearable device that can be worn by the user. For example, the electronic device could be AR glasses and / or a head-mounted display (HMD).

[0003] The above information is provided as relevant technology for the purpose of aiding understanding of this disclosure. No argument or decision is made regarding whether any of the above descriptions can be applied as prior art in connection with this disclosure. Summary of the Invention

[0004] Technical solution According to an embodiment, the wearable device may include a camera, a memory for storing instructions, and a processor for executing the instructions. The processor may be configured to: acquire an image of an external space including the wearable device using the camera. The processor may be configured to: identify portions of the image associated with reflectors of reflected light based on object recognition of the image. The processor may be configured to: provide a virtual space at least partially mapped to the external space by executing a software application using information based on whether each feature point of the image is included in the portion.

[0005] According to an embodiment, a method using a wearable device may include: acquiring an image of an external space including the wearable device using a camera of the wearable device. The method may include: identifying portions of the image associated with reflectors of reflected light based on object recognition of the image. The method may include: providing a virtual space at least partially mapped to the external space by performing a software application using information based on whether each feature point of the image is included in the portion.

[0006] According to an embodiment, the wearable device may include a camera, a memory for storing instructions, and a processor for executing instructions. The processor may be configured to: identify an external space including the wearable device based on a first feature point in an image obtained using the camera. The processor may be configured to: identify a second feature point having coordinate values ​​corresponding to the first external space and identified based on the camera, based on identifying the wearable device in a first external space or a second external space including a reflector, located in a first external space adjacent to the second external space. The processor may be configured to: obtain information associated with the second external space by using the first feature point and one or more third feature points, based on identifying the wearable device in the first or second external space, located in the second external space.

[0007] According to embodiments, a method for using a wearable device may include: identifying an external space including the wearable device based on a first feature point in an image obtained using a camera of the wearable device. The method may include: identifying a second feature point having coordinate values ​​corresponding to the first external space and identified based on the wearable device being included in a first external space or a second external space including a reflector, and being located in a first external space adjacent to the second external space. The method may include: obtaining information associated with the first external space based on a fourth feature point, one of the first and second feature points, wherein the fourth feature point is different from one or more third feature points associated with a reflector. The method may include: obtaining information associated with the second external space by using the first feature point and one or more third feature points, based on the wearable device being identified in the first or second external space and located in a second external space. Attached Figure Description

[0008] Figure 1 An embodiment of a wearable device for obtaining information about the external space is shown.

[0009] Figure 2a and Figure 2b An example of a block diagram of a wearable device according to an embodiment is shown.

[0010] Figure 3 An example flowchart of a wearable device according to an embodiment is shown.

[0011] Figure 4 An example of the operation of a wearable device for identifying portions of an image associated with a reflector is shown.

[0012] Figure 5a and Figure 5b An example of how a wearable device operates to identify the portion associated with a reflector in each of sequentially acquired images is shown.

[0013] Figure 6a , Figure 6b and Figure 6c An example of how a wearable device identifies its position in an external space including a reflector is shown.

[0014] Figure 7 An example of a user interface (UI) displayed by a wearable device that identifies reflectors is shown.

[0015] Figure 8 An example flowchart of a wearable device according to an embodiment is shown.

[0016] Figure 9a An example of a perspective view of a wearable device according to an embodiment is shown.

[0017] Figure 9b Examples of one or more hardware components placed in a wearable device according to embodiments are shown.

[0018] Figures 10a to 10b An example of the exterior of a wearable device according to this configuration is shown. Detailed Implementation

[0019] In the following description, various embodiments of this document will be described with reference to the accompanying drawings.

[0020] The various embodiments and terminology used in this document are not intended to limit the technology described herein to the specific embodiments, and should be understood to include various modifications, equivalents, or alternatives to the corresponding embodiments. Reference numerals may be used for similar components in the description of the drawings. Singular expressions may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as “A or B,” “at least one of A and / or B,” “A, B, or C,” or “at least one of A, B, and / or C” may include all possible combinations of items listed together. Expressions such as “first,” “second,” “first,” or “second” may modify the corresponding component, regardless of order or importance, and are used only to distinguish one component from another, but do not limit the corresponding component. When (e.g., a first) component is referred to as “connected (functionally or communicatively)” or “accessible” to another (e.g., a second) component, that component may be directly connected to the other component, or may be connected via another component (e.g., a third component).

[0021] The term "module" as used in this document can include a unit configured with hardware, software, or firmware, and is used interchangeably with terms such as logic, logic block, component, or circuit. A module can be a component of a whole configuration or the smallest unit or part thereof that performs one or more functions. For example, a module may be configured with an application-specific integrated circuit (ASIC).

[0022] Figure 1 An embodiment of a wearable device 101 for obtaining information about external space is shown. The wearable device 101 may include a head-mounted display (HMD) that can be worn on the head of a user 110. The wearable device 101 may be referred to as a head-mounted device (HMD), head-mounted electronics, eyeglasses-type electronics, video see-through (or visible see-through) (VST) device, extended reality (XR) device, virtual reality (VR) device, and / or augmented reality (AR) device. Although the appearance of the wearable device 101 in the form of eyeglasses is shown, the embodiment is not limited thereto. Reference will be made to... Figure 2a and / or Figure 2b An example of the hardware configuration included in wearable device 101 is described exemplarily. (Refer to...) Figure 9a , Figure 9b , Figure 10a and / or Figure 10b This describes a structural example of a wearable device 101 that can be worn on the head of a user 110. The wearable device 101 may be referred to as an electronic device. For example, the electronic device may form an HMD by being coupled with an accessory (e.g., a strap) that will be attached to the user's head.

[0023] According to an embodiment, wearable device 101 can perform functions associated with augmented reality (AR) and / or mixed reality (MR). For example, when user 110 is wearing wearable device 101, wearable device 101 may include at least one lens positioned adjacent to user 110's eyes. Wearable device 101 can combine ambient light passing through the lens with light emitted from a display of wearable device 101. The display area of ​​the display may be formed within the lens through which the ambient light passes. Because wearable device 101 combines ambient light and light emitted from the display, user 110 can view an image that blends real objects perceived by ambient light and virtual objects formed by light emitted from the display. The aforementioned augmented reality, mixed reality, and / or virtual reality may be referred to as extended reality (XR).

[0024] According to an embodiment, wearable device 101 can perform functions associated with video perspective (or visible perspective) (VST) and / or virtual reality (VR). For example, when user 110 is wearing wearable device 101, wearable device 101 may include a housing covering user 110's eyes. In this state, wearable device 101 may include a display placed on a first surface of the housing facing the eyes. Wearable device 101 may include a camera placed on a second surface opposite the first surface. By using the camera, wearable device 101 can acquire images and / or videos representing ambient light. Wearable device 101 can allow user 110 to perceive ambient light through the display by outputting images and / or videos on the display placed on the first surface. The display area (or active area) of the display placed on the first surface may be formed by one or more pixels included in the display. Wearable device 101 can allow user 110 to perceive virtual objects and real objects perceived by ambient light by combining virtual objects with images and / or videos output through the display.

[0025] According to an embodiment, the wearable device 101 can identify or recognize its position (or orientation) and / or direction (direction) based on images (and / or videos) obtained (or acquired) using a camera. (See also...) Figure 3 and / or Figure 8 An exemplary operation of the wearable device 101 based on image recognition location is described. (Refer to...) Figure 1 Exemplary images 140 and 150 obtained by a camera included in wearable device 101 are shown. By using a camera positioned in an oriented direction (e.g., forward direction) of wearable device 101, wearable device 101 can obtain images of external space. When worn by user 110, wearable device 101 can obtain images 140 and 150 by controlling the camera. Image 140 can be obtained when user 110 wearing wearable device 101 looks in direction dh1. Image 150 can be obtained when user 110 wearing wearable device 101 looks in direction dh2.

[0026] Reference Figure 1 This illustrates an exemplary environment in which a wearable device 101 is placed in an external space including a reflector 130, such as a mirror. The reflector 130 may include materials for reflecting ambient light (e.g., glass, acrylic, metal, and / or liquid). Embodiments are not limited thereto, and the reflector 130 may include electronic devices (such as smart mirrors) containing a selfie camera. The reflector 130 may have relatively high reflectivity. The reflector 130 may include a surface in which a material is uniformly distributed. Figure 1 In the exemplary environment, incident light on the surface of reflector 130 can be reflected at least partially toward wearable device 101. (Refer to...) Figure 1The portions of images 140 and 150 corresponding to reflector 130 can represent reflected light propagating toward wearable device 101.

[0027] Wearable device 101 can identify one or more external objects associated with an image (e.g., images 140 and 150) based on object recognition. For example, in image 140, wearable device 101 can identify wall surfaces 120 and 122 corresponding to visual objects 141 and 142, respectively. In image 140, wearable device 101 can identify a flowerpot 132 placed in an external space based on visual object 144. In image 140, wearable device 101 can identify a reflector 130 based on visual object 143. Based on visual objects 141, 142, 143, and 144 segmented from image 140, wearable device 101 can identify the location of external objects (e.g., wall surfaces 120 and 122, flowerpot 132, and / or reflector 130). Similarly, based on visual objects 151, 152, 153, and 154 segmented from image 150, wearable device 101 can identify the positions of external objects (e.g., wall surfaces 120 and 122, flowerpot 132, and / or reflector 130). The positions of the external objects identified by wearable device 101 correspond to the relative positions of the external objects with respect to wearable device 101. For example, visual object 143 will be referenced... Figure 4 This describes the operation of the wearable device 101 in recognizing the portion of image 140 associated with reflector 130. (Refer to...) Figure 7 An example of a user interface (UI) displayed by a wearable device 101 that identifies a reflector 130 is described.

[0028] For example, reflector 130 may reflect light from an external object different from reflector 130 toward wearable device 101. For example, user 110 wearing wearable device 101 adjacent to reflector 130 may be captured in the portion of image 140 corresponding to reflector 130. Similarly, user 110 wearing wearable device 101 may be captured in the portion of image 150 corresponding to reflector 130. Wearable device 101 may calculate the position and / or orientation of wearable device 101 more accurately in each of images 140 and 150 based on visual objects 143 and 153 determined to correspond to reflector 130. For example, wearable device 101 may identify whether each feature point in image 140 corresponds to reflected light from reflector 130. Wearable device 101 may determine whether to use each feature point to calculate the position and / or orientation of wearable device 101 based on whether each feature point in image 140 corresponds to reflected light from reflector 130. (See reference...) Figure 5a , Figure 5b , Figure 6a , Figure 6b and / or Figure 6cThe description describes the operation of the wearable device 101 in determining whether each of the feature points corresponds to reflected light.

[0029] In embodiments, the position and / or orientation of the wearable device 101, calculated by the wearable device 101, can be used to execute software applications on the wearable device 101 (e.g., software applications for virtual reality). For example, the wearable device 101 may execute the software application based on information indicating whether each feature point in the image 140 is included in the portion of the image 140 associated with the reflector 130. The wearable device 101 executing the software application may display, based on this information, a virtual space provided by the software application and at least partially mapped to (or coupled to) external space.

[0030] The operation of calculating the position and / or orientation of wearable device 101 in relation to tracking the position and / or orientation of the head of user 110 wearing wearable device 101 may be referred to as head tracking (HeT) (or head tracking function). For example, wearable device 101 may identify the position ph of user 110's head and / or wearable device 101 based on at least one of images 140 and 150. Wearable device 101 may identify each of the orientations dh1 and dh2 of head and wearable device 101 at each time point in capturing images 140 and 150. While displaying a virtual space mapped to external space, wearable device 101, upon acquiring image 140, may display one or more virtual objects set to be placed in orientation dh1. If user 110, who is looking in orientation dh1, rotates their head toward orientation dh2, wearable device 101 may display another virtual object set to be placed in orientation dh2 based on acquired image 150. Based on head tracking, wearable device 101 can display a screen (e.g., an AR and / or VR-based screen) that is synchronized with the movement of user 110's head.

[0031] As described above, the wearable device 101 according to the embodiment can identify at least a portion of an image associated with a reflector 130 reflecting light based on object recognition of the image (e.g., images 140 and 150). For example, the wearable device 101 can obtain information including at least one of the shape, position, or size of the portion in the image. The at least portion of the image associated with the reflector 130 may be referred to as a reflector region. The wearable device 101 can more accurately calculate the position and / or orientation of the wearable device 101 based on the positional relationship between feature points of the image and the reflector region.

[0032] In the following text, reference will be made to Figure 2a and / or Figure 2b An example of a hardware configuration for head tracking included in wearable device 101 is described.

[0033] Figure 2a and Figure 2b An example block diagram of a wearable device 101 according to an embodiment is shown. (Refer to...) Figure 1 The described wearable device 101 may include components made of Figure 2a At least one of the hardware components of the wearable device 101, which is distinguished by different frames of 2B.

[0034] Reference Figure 2a The wearable device 101 according to an embodiment may include at least one of a processor 210, a memory 215, a display 220, a camera 225, a sensor 230, or a communication circuit 235. The processor 210, memory 215, display 220, camera 225, sensor 230, and / or communication circuit 235 may be electronically and / or operatively coupled to each other via electronic components such as a communication bus 202. The type and / or number of hardware components included in the wearable device 101 are not limited to... Figure 2a Those shown. For example, wearable device 101 may only include those shown. Figure 2a Some of the hardware components shown.

[0035] The processor 210 of the wearable device 101 according to an embodiment may include hardware components for processing data based on one or more instructions. The hardware components for processing data may include, for example, an arithmetic and logic unit (ALU), a field-programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In an embodiment, the wearable device 101 may include one or more processors. The processor 210 may have a multi-core processor architecture, such as dual-core, quad-core, hexa-core, and / or octa-core.

[0036] The memory 215 of the wearable device 101 according to an embodiment may include hardware components for storing data and / or instructions input to and / or output from the processor 210. The memory 215 may include, for example, volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM)). The volatile memory may include at least one of, for example, dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include at least one of, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, optical disk, and embedded multimedia card (eMMC). In embodiments, the memory 215 may be referred to as a storage device.

[0037] In an embodiment, the display 220 of the wearable device 101 can display information to the user of the wearable device 101 (e.g., Figure 1User 110) outputs visual information (e.g., Figure 7 (Screen 710). For example, display 220 can output visual information to a user via a processor 210 controlled by circuitry including a graphics processing unit (GPU). Display 220 may include a flexible display, a flat panel display (FPD), and / or electronic paper. FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light-emitting diodes (LEDs). LED may include organic LEDs (OLEDs). Embodiments are not limited thereto, and for example, where wearable device 101 includes a lens for transmitting external light (or ambient light), display 220 may include a projector (or projection assembly) for projecting light onto the lens. In embodiments, display 220 may be referred to as a display panel and / or display module.

[0038] In an embodiment, the camera 225 of the wearable device 101 may include one or more optical sensors (e.g., charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors) that generate electrical signals indicating the color and / or brightness of light. The camera 225 may be referred to as an image sensor and may be included in... Figure 2a The sensor 230 is included in the camera 225. Multiple optical sensors included in the camera 225 can be arranged in a two-dimensional array. The camera 225 can generate two-dimensional frame data corresponding to the light arriving at the optical sensors of the two-dimensional array by substantially simultaneously acquiring electrical signals from each of the multiple optical sensors. For example, photographic data captured using the camera 225 can refer to two-dimensional frame data obtained from the camera 225. For example, video data captured using the camera 225 can refer to a sequence of multiple two-dimensional frame data obtained from the camera 225 according to the frame rate. The camera 225 may also include a flash positioned in the direction in which the camera 225 receives light, for outputting light in that direction.

[0039] The wearable device 101 according to an embodiment may include a plurality of cameras positioned in different directions, as an example of camera 225. (See also...) Figure 2a The camera 225 included in the wearable device 101 may include a gaze tracking camera 225-1 and / or an outward-facing camera 225-2. The gaze tracking camera 225-1 may be positioned facing at least one of the eyes of the user wearing the wearable device 101. The processor 210 may identify the user's gaze direction by using images and / or video obtained from the gaze tracking camera 225-1. The gaze tracking camera 225-1 may include an infrared (IR) sensor. The gaze tracking camera 225-1 may be referred to as an eye sensor, a gaze tracker, and / or an eye tracker.

[0040] In an embodiment, the outward-facing camera 225-2 may be positioned facing forward (e.g., in the direction that the two eyes may be pointing) towards the user wearing the wearable device 101. The direction in which the outward-facing camera 225-2 points is not limited to the forward direction. The outward-facing camera 225-2 may be placed on any surface different from the surface of the wearable device 101 that is in contact with the user when worn. Using the images and / or video obtained from the outward-facing camera 225-2, the processor 210 may identify external objects (e.g., Figure 1 The reflector 130). The embodiments are not limited thereto, and the processor 210 may identify the position, shape and / or posture (e.g., gesture) of the hand based on images and / or videos obtained from the outward-facing camera 225-2.

[0041] According to an embodiment, the sensor 230 of the wearable device 101 can generate electronic information from non-electronic information associated with the wearable device 101, which can be processed and / or stored by the processor 210 and / or memory 215 of the wearable device 101. This information may be referred to as sensor data. The sensor 230 may include a Global Positioning System (GPS) sensor, an image sensor, an illumination sensor, and / or a Time-of-Flight (ToF) sensor (or a ToF camera) for detecting the geographical location of the wearable device 101.

[0042] In an embodiment, sensor 230 may include an inertial measurement unit (IMU) for detecting physical motion of wearable device 101. An accelerometer, a geomagnetic sensor, a gyroscope sensor, or a combination thereof may be referred to as an IMU. The accelerometer may output an electrical signal indicating gravitational acceleration and / or acceleration of each of a plurality of axes (e.g., x-axis, y-axis, and z-axis) perpendicular to each other and based on a preset origin of wearable device 101. The gyroscope sensor may output an electrical signal indicating the angular velocity of each of the plurality of axes. The gyroscope sensor may be referred to as an angular velocity sensor. The geomagnetic sensor may output an electrical signal indicating the magnitude of a magnetic field formed in wearable device 101 along each of the plurality of axes (e.g., x-axis, y-axis, and / or z-axis). For example, the accelerometer, gyroscope sensor, and / or geomagnetic sensor may repeatedly output sensor data including acceleration, angular velocity, and / or magnetic field magnitude corresponding to the number of the plurality of axes based on a preset period (e.g., 1 millisecond).

[0043] In an embodiment, the communication circuitry 235 of the wearable device 101 may include circuitry for supporting the transmission and / or reception of electrical signals between the wearable device 101 and an external electronic device. The communication circuitry 235 may include at least one of, for example, a modulator and demodulator (modem), an antenna, or an optical / electrical (O / E) converter. The communication circuitry 235 may support the transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, Local Area Network (LAN), Wide Area Network (WAN), Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), ZigBee, Long Term Evolution (LTE), 5G New Radio (NR), 6G, and / or above-6G. In an embodiment, the communication circuitry 235 may be referred to as a communication processor and / or a communication module.

[0044] According to an embodiment, the memory 215 of the wearable device 101 may store one or more instructions (or commands) that indicate data to be processed, calculations to be performed, and / or operations to be executed by the processor 210 of the wearable device 101. The set of one or more instructions may be referred to as firmware, operating system, process, routine, subroutine, and / or software application (hereinafter referred to as application). For example, when executing a set of multiple instructions distributed in the form of an operating system, firmware, driver, and / or application, the wearable device 101 and / or processor 210 may perform... Figure 3 and / or Figure 8 At least one of the operations. In the following, an application installed in wearable device 101 may mean that one or more instructions provided in the form of an application are stored in memory 215, and that one or more applications are stored in a format executable by processor 210 (e.g., a file with an extension preset by the operating system of wearable device 101). As an example, an application may include programs and / or libraries associated with services provided to the user.

[0045] Reference Figure 2a The programs installed in the wearable device 101 can be categorized into any of the different layers, including the application layer 240, the frame layer 250, and / or the hardware abstraction layer (HAL) 280, based on their target. For example, in the hardware abstraction layer 280, programs (e.g., modules or drivers) designed to target the hardware of the wearable device 101 (e.g., display 220, camera 225, sensor 230, and / or communication circuitry 235) can be categorized. The frame layer 250 can be referred to as the XR frame layer in terms of including one or more programs for providing extended reality (XR) services. For example, Figure 2a The layers shown are logically (or for ease of illustration) divided, and this may not mean that the address space of memory 215 is divided by layers.

[0046] For example, in frame layer 250, programs designed to target at least one of hardware abstraction layer 280 and / or application layer 240 (e.g., position tracker 271, spatial awareness unit 272, pose tracker 273, gaze tracker 274, and / or face tracker 275) can be categorized. Programs categorized as frame layer 250 can provide executable application programming interfaces (APIs) based on another program.

[0047] For example, in application layer 240, programs designed to target users of wearable device 101 can be categorized. Examples of programs categorized as application layer 240 include extended reality (XR) system user interface (UI) 241 and / or XR application 242, but embodiments are not limited to this. For example, programs categorized as application layer 240 (e.g., software applications) can perform functions supported by programs categorized as framework layer 250 by calling application programming interfaces (APIs).

[0048] For example, based on the execution of the XR system UI 241, the wearable device 101 can display one or more visual objects on the display 220 to perform interactions with the user to utilize the virtual space. Visual objects can refer to objects that can be deployed on a screen for information transmission and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. Visual objects can be referred to as visual guides, virtual objects, visual elements, UI elements, view objects, and / or view elements. The wearable device 101 can provide the user with the functions available in the virtual space based on the execution of the XR system UI 241.

[0049] Reference Figure 2a This illustrates that a lightweight renderer 243 and / or an XR plugin 244 are included in the XR system UI 241, but are not limited thereto. For example, based on the XR system UI 241, the processor 210 may execute the lightweight renderer 243 and / or the XR plugin 244 in the frame layer 250.

[0050] For example, wearable device 101 may obtain resources (e.g., APIs, system processes, and / or libraries) for defining, generating, and / or executing a rendering pipeline in which partial modifications are permitted, based on the execution of lightweight renderer 243. Lightweight renderer 243 may be referred to as a lightweight rendering pipeline in defining a rendering pipeline in which partial modifications are permitted. Lightweight renderer 243 may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, wearable device 101 may obtain resources (e.g., APIs, system processes, and / or libraries) for defining, generating, and / or executing the entire rendering pipeline based on the execution of XR plugin 244. XR plugin 244 may be referred to as an open XR native client in defining (or setting up) the entire rendering pipeline.

[0051] For example, wearable device 101 may display a screen indicating at least a portion of the virtual space on display 220 based on the execution of XR application 242. XR plugin 244-1 included in XR application 242 may include instructions supporting functions similar to those of XR plugin 244 in XR system UI 241. Descriptions that overlap with those of XR plugin 244 may be omitted in the description of XR plugin 244-1. Wearable device 101 may invoke the execution of virtual space manager 251 based on the execution of XR application 242.

[0052] According to an embodiment, wearable device 101 may provide virtual space services based on the execution of virtual space manager 251. For example, virtual space manager 251 may include a platform for supporting virtual space services. Based on the execution of virtual space manager 251, wearable device 101 may identify a virtual space formed based on the user's location indicated by data obtained through sensor 230, and may display at least a portion of the virtual space on display 220. Virtual space manager 251 may be referred to as compositional rendering manager (CPM).

[0053] For example, the virtual space manager 251 may include a runtime service 252. As an example, the runtime service 252 may be referred to as an OpenXR runtime module (or OpenXR runtime program). Based on the execution of the runtime service 252, the wearable device 101 may perform at least one of user pose prediction, frame timing, and / or spatial input functions. As an example, the wearable device 101 may render virtual space services to the user based on the execution of the runtime service 252. For example, based on the execution of the runtime service 252, virtual space-related functions executable by the application layer 240 may be supported.

[0054] For example, the virtual space manager 251 may include a pass-through manager 253. Based on the execution of the pass-through manager 253, the wearable device 101 may display a screen indicating the virtual space on the display 220 while simultaneously displaying another screen indicating the real space obtained through the outward-facing camera 225-2 on at least a portion of the screen.

[0055] For example, the virtual space manager 251 may include an input manager 254. Based on the execution of the input manager 254, the wearable device 101 can recognize data (e.g., sensor data) obtained by executing one or more procedures included in the perception service layer 270. The wearable device 101 can use the obtained data to recognize user input associated with the wearable device 101. User input may be associated with user movements (e.g., gestures), gaze, and / or voice recognized by the sensor 230.

[0056] For example, the Perception Abstraction Layer 260 can be used for data exchange between the Virtual Space Manager 251 and the Perception Service Layer 270. In its use for data exchange between the Virtual Space Manager 251 and the Perception Service Layer 270, the Perception Abstraction Layer 260 can be referred to as an interface. As an example, the Perception Abstraction Layer 260 can be referred to as OpenPX. The Perception Abstraction Layer 260 can be used for both perception clients and perception services.

[0057] According to an embodiment, the perception service layer 270 may include one or more programs for processing data obtained from the sensor 230 and / or the camera 225. The one or more programs may include at least one of a position tracker 271, a spatial sensing unit 272, a pose tracker 273, a gaze tracker 274, and / or a face tracker 275. The type and / or number of the one or more programs included in the perception service layer 270 are not limited to... Figure 2a Those shown in the image.

[0058] For example, based on the execution of position tracker 271, wearable device 101 can use sensor 230 to identify the posture of wearable device 101. Based on the execution of position tracker 271, wearable device 101 can identify its 6-DOF posture using data obtained using outward-facing camera 225-2 and / or IMU (e.g., gyroscope sensor, accelerometer sensor, and / or geomagnetic sensor). Position tracker 271 may be referred to as a head tracking (HeT) module (or head tracker or head tracking program).

[0059] For example, based on the execution of the spatial sensing unit 272, the wearable device 101 can obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device 101 (or the user of the wearable device 101). Based on the execution of the spatial sensing unit 272, the wearable device 101 can reconstruct its surrounding environment in three dimensions using data obtained using the outward-facing camera 225-2. Based on the execution of the spatial sensing unit 272, the wearable device 101 can identify at least one of planes, inclinations, and steps according to the reconstructed surrounding environment in three dimensions. The spatial sensing unit 272 may be referred to as a scene understanding (SU) module (or scene understanding program).

[0060] For example, wearable device 101 may identify (or sense) the user's posture and / or hand posture based on the execution of posture tracker 273. As an example, based on the execution of posture tracker 273, wearable device 101 may identify the user's posture and / or hand posture using data obtained from external camera 225-2. As an example, based on the execution of posture tracker 273, wearable device 101 may identify the user's posture and / or hand posture based on data (or images) obtained using external camera 225-2. Posture tracker 273 may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or posture tracking module.

[0061] For example, wearable device 101 may identify (or track) the movement of the user's eyes based on the execution of gaze tracker 274. As an example, wearable device 101 may identify the movement of the user's eyes by using data obtained from gaze tracking camera 225-1 when gaze tracker 274 is executed. Gaze tracker 274 may be referred to as an eye tracking (ET) module (or eye tracking program) and / or gaze tracking module.

[0062] For example, the perception service layer 270 of wearable device 101 may also include a face tracker 275 for tracking a user's face. For example, wearable device 103 may identify (or track) movement of the user's face and / or the user's facial expressions based on the execution of face tracker 275. Based on the execution of face tracker 275, wearable device 101 may estimate the user's facial expressions based on movement of the user's face. As an example, based on the execution of face tracker 275, wearable device 101 may identify movement of the user's face and / or the user's facial expressions based on data (e.g., images) obtained using sensor 230 (e.g., an image sensor facing at least a portion of the user's face).

[0063] According to an embodiment, wearable device 101 can identify its position in external space as indicated by information obtained from camera 225 and / or sensor 230. Based on this position, wearable device 101 can provide at least a portion of a virtual space mapped to (or coupled to) external space. For example, based on this position, wearable device 101 can determine its position in the virtual space. The determined position in the virtual space can correspond to a reference point (e.g., a viewpoint) used to display a portion of the virtual space that will be displayed via display 220. The processor 210 of wearable device 101 can execute... Figure 2a The position tracker 271 executes one or more instructions for identifying a position in external space.

[0064] Reference Figure 2b The instructions included in the location tracker 271 can be functionally categorized into a sensor manager 291, a position and orientation processor 293, a feature point extractor 294, a reflector detector 295, and / or a map generator 296. These instructions can be executed by the processor 210 of the wearable device 101.

[0065] Based on the execution of sensor manager 291, wearable device 101 can store sensor data of sensor 230 in memory 215 of wearable device 101. Based on the execution of sensor manager 291, wearable device 101 can manage the sensor data stored in memory 215. For example, wearable device 101 can store and / or delete sensor data (e.g., angular velocity, acceleration, and / or magnetic field direction) obtained from IMU in memory 215. Wearable device 101 can obtain integrated values ​​of angular velocity, acceleration, and / or magnetic field direction included in the sensor data obtained from IMU. The values ​​obtained by wearable device 101 can indicate the orientation and / or position of wearable device 101 in external space. By integrating image and / or sensor data accumulated over a specific time period, wearable device 101 can obtain vector data corresponding to that time period. Vector data can be obtained to reduce the amount of computation required to identify the position and / or orientation of wearable device 101. Wearable device 101 can obtain vector data in response to calls to APIs used for integrating sensor data. The embodiments are not limited thereto, and the wearable device 101 may store the image and / or sensor data accumulated during a specific time period in the memory 215 of the wearable device 101.

[0066] Based on the execution of feature point extractor 294, wearable device 101 can obtain images (e.g., frame images of video) from camera 225. Figure 1Images 140 and 150 identify one or more feature points. Feature points (or keypoints) in an image can be points in the image used to examine features of the image (or objects captured by the image). For example, by comparing feature points included in each of the different images, wearable device 101 can identify external objects commonly captured in the images. By comparing feature points in different images, wearable device 101 can identify the landmarks of external objects commonly captured in the images. Based on the feature points included in the images, wearable device 101 can calculate the three-dimensional spatial coordinates of the external object associated with the feature points. Based on the feature points included in the images, wearable device 101 can determine the images as keyframes.

[0067] Based on the execution of feature point extractor 294, wearable device 101 can perform calculations instructed by algorithms such as Scale Invariant Feature Transform (SIFT), Harris Corner Detection, Faster-Short Test Feature (FAST), and / or Oriented Fast and Rotated Brief (ORB). Based on this calculation, wearable device 101 can extract feature points from an image, or can perform a comparison (or matching) of feature points included in an image. By comparing feature points included in images obtained at different time points and / or locations, wearable device 101 can determine the three-dimensional coordinates of each feature point. Wearable device 101, which identifies feature points commonly included in multiple images, can obtain information indicating the time point in which the feature points (or multiple images) were identified.

[0068] Based on the execution of feature point extractor 294, wearable device 101 can obtain one or more feature points from images acquired by camera 225 for calculating the position and / or orientation of wearable device 101. Wearable device 101, having acquired multiple images, can select and / or determine keyframes in the multiple images based on the number of feature points extracted from each of the multiple images. For example, an image in which a relatively large number (or a number greater than a preset number) of feature points is detected can be selected as a keyframe. For example, by comparing feature points commonly included in different images, wearable device 101 can select an image with a relatively small number of feature points common to another image acquired at a time point prior to a specific time point of image recognition. Wearable device 101 can store information 292 in its memory 215 indicating the feature points extracted from the images and / or the results of keyframe selection.

[0069] Based on the execution of the reflector detector 295, the wearable device 101 can identify reflectors (e.g., in the image) in the image. Figure 1The wearable device 101, which identifies at least a portion corresponding to the reflector 130, can calculate the position and / or size of that at least portion in the image. The wearable device 101 can then calculate the position (e.g., relative position with respect to the wearable device 101) and / or size of the reflector based on that at least portion.

[0070] The wearable device 101 according to an embodiment may execute a computational model for extracting reflectors from an image based on a reflector detector 295. The computational model may include an artificial neural network that simulates human reasoning activities. For example, the computational model may include an artificial neural network (such as a convolutional neural network (CNN)) for processing an image consisting of a set of pixels arranged in two dimensions. Embodiments are not limited thereto, and the computational model may include a feedforward neural network (FNN), a long short-term memory (LSTM), and / or a recurrent neural network (RNN). The artificial neural network simulated based on the reflector detector 295 may be a pre-trained model based on supervised and / or unsupervised learning.

[0071] Based on the execution of reflector detector 295, wearable device 101 can obtain information (e.g., a probability map) including the probability that each pixel in the image corresponds to a reflector. Based on this information, wearable device 101 can identify portions of the image corresponding to reflectors (e.g., portions with quadrilateral and / or circular shapes). Based on the execution of feature point extractor 294, wearable device 101, which identifies one or more feature points from the image, can determine whether each of the one or more feature points corresponds to a reflector based on the probability obtained by reflector detector 295. Wearable device 101 can store parameters (e.g., flags and / or probabilities) indicating whether each of the feature points included in information 292 corresponds to a reflector in information 292 stored in memory.

[0072] Based on the execution of map generator 296, wearable device 101 can generate map information about the external space in which it is located by using information 292 indicating feature points and / or keyframes. Wearable device 101 can generate map information (e.g., local maps) about each of different external spaces with dimensions smaller than a preset size. Wearable device 101 can merge map information about each of the external spaces. By comparing images acquired in real time using camera 225 with the map information generated by map generator 296, wearable device 101 can determine whether its current location corresponds to a point in the external space and / or the map information. When the current location corresponds to a point in the map information, wearable device 101 can recognize a revisit to that point.

[0073] Map information (e.g., a local map) generated based on the execution of map generator 296 may include information about feature points and / or keyframes used to generate the map information. The map information generated based on the execution of map generator 296 may be stored in memory 215 of wearable device 101. Based on the execution of map generator 296, wearable device 101 may manage (e.g., generate and / or delete) the map information stored in memory 215. In embodiments, map information may be units storing feature points and / or keyframes obtained (or extracted) based on feature point extractor 294. By comparing different local maps, wearable device 101 may determine whether to merge local maps. For example, if the similarity between local maps is greater than a preset threshold, wearable device 101 may merge local maps. Merging local maps by wearable device 101 may mean managing (e.g., sending, copying, and / or deleting) the feature points and / or keyframes corresponding to each local map as a single unit.

[0074] Based on the execution of map generator 296, wearable device 101 can determine its current position according to images recognized in real time from camera 225. For example, when feature points corresponding to specific map information generated by map generator 296 are identified, wearable device 101 can determine the position corresponding to the specific map information as its current position. For example, when an image similar to a keyframe stored by map generator 296 is obtained from camera 225, wearable device 101 can determine the position corresponding to the keyframe as its current position. Local maps spaced apart from the current position can be removed from the volatile memory of wearable device 101 or stored in non-volatile memory.

[0075] Based on the execution of the position and orientation processor 293, the processor 210 of the wearable device 101 can calculate the position and / or orientation of the wearable device 101. The wearable device 101 can calculate its position and / or orientation based on at least one of information 292 including feature points and / or keyframes and / or information obtained by the reflector detector 295. For example, the wearable device 101 can use information obtained by the position tracker 271 to calculate its position and / or orientation. For example, based on the probability of each feature point corresponding to a reflector obtained from the reflector detector 295, the wearable device 101 can determine whether to use the feature points to calculate its position and / or orientation. The wearable device 101 can update the information 292 including feature points and keyframes based on its position and / or orientation. For example, the wearable device 101 can prevent feature points corresponding to reflectors from being used by the map generator 296 (e.g., not used to generate map information).

[0076] Based on the execution of position tracker 271, wearable device 101 can obtain information for executing software applications (e.g., software applications included in application layer 240). For example, wearable device 101, which identifies a preset API included in the software application, can calculate the position and / or orientation of wearable device 101 requested by the API by executing position tracker 271. The API may include parameters for requesting the position and / or orientation of wearable device 101 at a specific point in time. The software application may include instructions for repeatedly invoking the preset API based on a preset period and / or frequency.

[0077] Based on a request invoked (or referenced) by a software application, wearable device 101 may execute position tracker 271. This request can be identified by an API. To identify the position and / or orientation of wearable device 101 at a first time point corresponding to the request, wearable device 101 may obtain information indicating the position and / or orientation at a second time point prior to the first time point. By combining the position and / or orientation at the second time point with sensor data (e.g., sensor data obtained from sensor 230) and images (e.g., images obtained from camera 225) from the time period between the second and first time points, wearable device 101 may calculate the position and / or orientation of wearable device 101 at the first time point. Wearable device 101 may invoke a function (or callback function, routine, subroutine, and / or method) of the software application indicated by the request based on the calculated position and / or orientation.

[0078] In one embodiment, wearable device 101 can calculate the position and / or orientation of wearable device 101 at a specific time point by executing position tracker 271 based on data indicating a specific time point (e.g., a timestamp). Wearable device 101 can also calculate the position and / or orientation of wearable device 101 based on sensor data accumulated over a time period including the specific time point.

[0079] The information obtained from the execution of the location tracker 271 and used to execute the software application is not limited to the position and / or orientation of the wearable device 101 described above. The wearable device 101 can provide a virtual space mapped to external space by executing the software application based on map information identified by the map generator 296. The map information may include a point cloud comprising one or more feature points identified from an image.

[0080] As described above, the wearable device 101 according to the embodiments may include system software (e.g., a position tracker 271) for obtaining information indicating the position and / or orientation of the wearable device 101 as information for executing software applications of the wearable device 101. For example, by executing instructions for a software application that recreates a virtual space based on the information, the wearable device 101 may display at least a portion of a virtual space coupled to the external space on a display 220. The wearable device 101 may identify reflectors (e.g., in the external space) included in the external space by executing the position tracker 271, which includes a reflector detector 295. Figure 1 (Reflector 130). The result of identifying the reflector can be used to compensate for errors in the position and / or orientation of the wearable device 101 caused by identifying an image including the reflector. Referring below... Figure 3 Exemplary operation of wearable device 101 based on reflector recognition is described.

[0081] Figure 3 An example flowchart of a wearable device according to an embodiment is shown. Figure 1 , Figure 2a and Figure 2b Wearable device 101 and / or Figure 2a The processor 210 can execute reference Figure 3 The described operation. Figure 3 At least one of the operations can be performed with Figure 2a and / or Figure 2b The location tracker 271 is associated with the operation of the wearable device.

[0082] In the following embodiments, each of the operations may be executed sequentially, but not necessarily in a sequential order. For example, the order of each of the operations may be changed, and at least two operations may be executed in parallel.

[0083] Reference Figure 3 In operation 310, the processor of the wearable device according to the embodiment can perform head tracking functionality while it is worn by a user. The processor can perform head tracking based on whether it is worn by a user (e.g., Figure 1 User 110) wears the device and switches between idle and active states. Figure 2a and / or Figure 2b The state of the position tracker 271. For example, based on sensors (e.g., Figure 2aThe processor, which detects the movement of the wearable device via sensor 230, can switch the state of position tracker 271 from idle to active. Embodiments are not limited to this, and wearable devices including attachable straps can identify the connection of the strap based on switching circuitry included in the strap. The wearable device identifying the connection of the strap can perform operation 310. For example, when worn by a user, the processor can initiate the execution of head tracking functionality in operation 310 by switching the state of position tracker 271 to active.

[0084] Reference Figure 3 In operation 320, the processor of the wearable device according to the embodiment can detect reflective objects based on images from the camera (e.g., Figure 1 The reflector 130). The camera operating 320 may include Figure 2a Camera 225 and / or outward-facing camera 225-2. The processor can be based on... Figure 2b The processor executes operation 320 by means of the reflector detector 295. Operation 320 may be executed based on a preset period or frequency. Operation 320 may be executed repeatedly (or periodically) based on a period and / or duration greater than the duration required to identify a reflector from a particular image. For example, the period for repeatedly executing operation 320 may be longer than the frame rate of the camera.

[0085] In an embodiment, the processor may repeatedly execute operation 320 based on a dynamically changing period. For example, the processor may select a keyframe from images sequentially output from the camera in the time domain and perform operation 320 on the selected keyframe. For example, the processor may execute operation 320 based on the number of feature points in the image. For example, the period of repeating operation 320 may be inversely proportional to the number of feature points extracted from the image. For example, the period of repeating operation 320 may be changed based on whether a reflector has been detected in the image. If no reflector is detected in the image during a preset duration, the processor may execute operation 320 based on a relatively long period. When a reflector is identified, the processor may reduce the period of repeating operation 320.

[0086] Reference Figure 3In operation 330, the processor of the wearable device according to the embodiment may obtain information associated with a portion of the image associated with a reflector. The processor, which detects the reflector based on operation 320, may perform operation 330. For example, the processor may obtain information about a portion (or region) in the image associated with a reflector. This information may include at least one of the following: the time point at which the reflector was identified (e.g., a timestamp), the type of reflector identified based on the image (e.g., class and / or category), the size of the reflector, the location in the image corresponding to the reflector (e.g., coordinates expressed in pixels), or an identifier (e.g., an ID) used to track the reflector. The information in operation 330 may include pixel-level values ​​(e.g., flags and / or probability values) indicating at least a portion of the image associated with a reflector.

[0087] Reference Figure 3 In operation 340, the processor of the wearable device according to the embodiment can adjust the attributes of feature points extracted from the image based on information. The feature points of operation 340 can be obtained based on the execution of feature point extractor 294. Attributes may include values ​​(e.g., flags and / or probability values) indicating whether a corresponding feature point corresponds to reflected light from a reflector. The image of operation 340 may correspond to the image of operation 330. The embodiment is not limited thereto, and the image of operation 340 may correspond to another image different from the image of operation 330.

[0088] For example, a processor that sequentially acquires multiple images in the temporal domain from a camera can extract feature points from an image at a first time point. A second time point, where feature point extraction is completed, may differ from a third time point, where the portion of the image associated with a reflector is identified. For example, the third time point may be a time point after the second time point. At the third time point, the processor can extract feature points from another image, different from the image at the first time point. In an embodiment, the processor can identify or search for feature points in an image and / or an image corresponding to the information of operation 330 based on the time point of the image corresponding to the information included in operation 330. The processor can change the attribute of each of the searched feature points based on whether the corresponding feature point is associated with a reflector. This attribute may include... Figure 2b Information 292.

[0089] Reference Figure 3 In operation 350, according to an embodiment, the processor of the wearable device can determine the position and / or orientation of the wearable device based on the attributes of feature points. Figure 2b The position and orientation processor 293 executes operation 350. The processor can perform operation 350 by using information stored in memory (e.g., ...). Figure 2b (Information associated with feature points and / or keyframes) to perform operation 350.

[0090] In an embodiment, the processor may be based on Figure 2b The execution of map generator 296 performs operation 350. For example, based on the execution of map generator 296, the processor may determine a revisit. A revisit may be determined based on whether the current location of the wearable device, as identified by the processor, matches the location of the map information obtained by map generator 296. To determine a revisit, the processor may use remaining feature points from a plurality of feature points, excluding one or more feature points having attributes associated with a reflector. Based on operation 350, the processor may execute an algorithm for determining a revisit, such as loop closure. The processor performing loop closure may not use at least one feature point indicated as associated with a reflector based on operation 340 for loop closure.

[0091] The position and / or orientation of the wearable device determined based on operation 350 can be provided to a software application executed by the wearable device 101. The processor can provide the position and / or orientation of operation 350 to the software application. The processor can display at least a portion of a virtual space associated with the position and / or orientation by executing the software application based on the position and / or orientation. Because the position and / or orientation of the wearable device is determined based on feature points having attributes adjusted according to reflectors, the processor can determine the position and / or orientation more accurately.

[0092] In the following text, reference will be made to Figure 4 This describes an exemplary operation of a wearable device that detects reflectors from an image based on operation 320.

[0093] Figure 4 An example of the operation of a wearable device 101 for identifying portions 430 of an image 140 associated with a reflector is shown. Figure 1 , Figure 2a and Figure 2b Wearable device 101 and / or Figure 2a The processor 210 can execute reference Figure 4 The operation of the wearable device 101 is described.

[0094] According to an embodiment, the wearable device 101 can access the camera 225 (e.g., Figure 2a The outward-facing camera 225-2) acquires image 140. The wearable device 101 may perform functions for processing image 140 based on position tracker 271. For example, the wearable device 101 may perform operations for identifying the portion 430 of image 140 corresponding to a reflector based on reflector detector 295.

[0095] Reference Figure 4A neural network 410 for detecting reflectors may be provided as part of a position tracker 271. The neural network 410 may include a set of parameters for defining the nodes included in multiple layers of the artificial neural network and the weights (or filters) between the nodes. The wearable device 101 may include software, hardware, or a combination thereof for operating the neural network 410. The software for operating the neural network 410 may include one or more programs and / or a set of instructions (e.g., a library) called by one or more programs to perform calculations associated with the multiple parameters. The hardware for operating the neural network 410 may include a CPU, a GPU, a neural processing unit (NPU), or a combination thereof.

[0096] Reference Figure 4 By using a neural network 410 over the input image 140, the wearable device 101 can identify the location of a portion 430 in the image 140 associated with a reflector. From the neural network 410 over the input image 140, the wearable device 101 can obtain reflector information 420. The reflector information 420 may include data indicating the location and / or size of the portion 430 in the image 140 corresponding to the reflector. (See also...) Figure 4 Including with reflectors (e.g., Figure 1 In an image 140 corresponding to a visual object 143 (reflector 130), the wearable device 101 can identify a portion 430 corresponding to the surface of the reflector. Reflected light from the reflector can be captured in portion 430 of image 140. The portion 430 in image 140 identified based on neural network 410 can correspond to or include the material (e.g., glass, metal, and / or organic compounds such as acrylic) encased by the frame of the mirror in the visual object 143 corresponding to the mirror.

[0097] The reflector information 420 generated based on the execution of the reflector detector 295 can be stored in the memory of the wearable device 101 (e.g., Figure 2a The reflector information 420 may be stored together with the image 140 in memory 215. The reflector information 420 may be used to determine whether feature points included in the image 140 are associated with a reflector. For example, among the feature points included in the image 140, one or more feature points included in the portion 430 identified by the reflector information 420 may have attributes indicating their association with a reflector. In this example, feature points included in another portion of the image 140 different from the portion 430 may have attributes indicating that they are not associated with a reflector (or attributes indicating that they are associated with an object different from a reflector).

[0098] In an embodiment, the size and / or position of the portion 430 indicated by the reflector information 420 can be used to determine whether feature points of the image 140 included in the portion 430 are used to identify the position of the wearable device 101. For example, if the ratio of the size of the portion 430 in the image 140 is greater than a preset ratio, the wearable device 101 can use the feature points located in the portion 430 to calculate the position of the wearable device 101. For example, if the ratio of the size of the portion 430 in the image 140 is less than a preset ratio, the wearable device 101 may discard the feature points located in the portion 430, or may not use them to calculate the position of the wearable device 101.

[0099] As described above, the wearable device 101 according to the embodiment can identify the portion 430 associated with a reflector from the image 140 using a neural network 410 provided together with the position tracker 271. The reflector information 420 obtained by the wearable device 101 may include data indicating the size, position, and / or shape of the portion 430. The wearable device 101 that identifies the portion 430 can obtain information indicating whether each of the feature points included in the image 140 is included in the portion 430 (e.g., Figure 3 (The attributes of operation 340). Wearable device 101 can use reflector information 420 corresponding to image 140 to perform head tracking based on image 140.

[0100] In the following text, reference will be made to Figure 5a and / or Figure 5b The following describes an exemplary operation in which a wearable device 101 performs head tracking based on images acquired sequentially in the time domain.

[0101] Figure 5a and Figure 5b An operational example of a wearable device 101 is shown for identifying portions associated with a reflector in each of the sequentially acquired images 521, 522, 523, and 524. Figure 1 , Figure 2a and Figure 2b Wearable device 101 and / or Figure 2a The processor 210 can execute reference Figure 5a and / or Figure 5b The operation of the wearable device 101 is described.

[0102] Reference Figure 5a and / or Figure 5b This illustrates an exemplary environment of a wearable device 101 worn by user 110 adjacent to a reflector 510 (e.g., a mirror with a rectangular shape). (Refer to...) Figure 5aAn exemplary case illustrates the directions dh1, dh2, dh3, and dh4 viewed by a user 110 moving from each of positions ph1, ph2, ph3, and ph4 along the path connecting positions ph1, ph2, ph3, and ph4. (Refer to...) Figure 5a This illustrates how the camera (e.g., from the wearable device 101) captures images of the head as it gradually changes direction from dh1 to dh4. Figure 2a Images 521, 522, 523 and 524 obtained by camera 225 and / or outward-facing camera 225-2.

[0103] Image 521, acquired at time t1 when user 110 at position ph1 is looking in direction dh1, may include portion 512 corresponding to reflector 510. Wearable device 101 may extract one or more feature points from image 521. At time t1+a, the operation of acquiring one or more feature points corresponding to image 521 can be completed. Figure 5a Image 531 may be an exemplary image that displays one or more feature points on image 521 to describe one or more feature points (e.g., markers in the form of quadrilaterals and / or triangles) extracted from image 521.

[0104] Image 522, acquired at time t2 when the user 110 at position ph2 is looking in direction dh2, may include a portion 512 corresponding to reflector 510. The shape of portion 512 may differ from the shape of portion 512 in image 521 acquired at time t1. At time t2+b, after time t2, wearable device 101 may acquire feature points from image 522. Image 532 may be an exemplary image used to describe the relationship between image 522 and the feature points acquired from image 522.

[0105] Image 523, acquired at time t3 when user 110 at position ph3 is looking in direction dh3, may include portion 512 corresponding to reflector 510. At time t3+c, after time t3, wearable device 101 may acquire feature points from image 523. Image 533 may be an exemplary image used to describe the relationship between image 523 and the feature points acquired from image 523.

[0106] Image 524, acquired at time t4 when the user 110 at position ph4 is looking in direction dh4, may include a portion 512 corresponding to reflector 510. At time t4+d, after time t4, wearable device 101 may acquire feature points from image 524. Image 534 may be an exemplary image used to describe the relationship between image 524 and the feature points acquired from image 524.

[0107] Wearable device 101 can identify at least one feature point commonly included in different images 531, 532, 533, and 534 by comparing feature points shown in images 531, 532, 533, and 534. For example, wearable device 101 can examine or identify feature points included in all images 531 and 532 by comparing feature points in each of images 531 and 532. Wearable device 101, which identifies feature points included in all images 531 and 532, can calculate the three-dimensional spatial coordinates corresponding to the feature points based on changes in the positions of the feature points in images 531 and 532. Based on the three-dimensional spatial coordinates (or changes in the three-dimensional spatial coordinates), wearable device 101 can calculate or identify the current position and / or current orientation of wearable device 101.

[0108] The time point at which wearable device 101 extracts feature points from a specific image (e.g., any one of images 521, 522, 523, and 524) and the time point at which wearable device 101 identifies the portion associated with the reflector can be independent of each other. (Refer to...) Figure 5a and / or Figure 5b The duration (e.g., a, b, c, and d) for extracting feature points from each of images 521, 522, 523, and 524 can be shorter than the interval (e.g., t2-t1) between the time points when each of images 521, 522, 523, and 524 is obtained.

[0109] Reference Figure 5b The wearable device 101 can identify the portion 512 associated with the reflector 510 in the image 521 at time t1+x, wherein feature point extraction of images 521, 522, and 523 at time points t1, t2, and t3 has been completed. For example, the wearable device 101 can identify a region 551 of feature points corresponding to the portion 512 in the feature points surrounding image 521. In the feature points of image 521, feature points associated with the reflector can be placed within region 551. Region 551 may have a polygonal shape connecting feature points (e.g., feature point f1) adjacent to the portion 512 associated with the reflector 510.

[0110] Reference Figure 5bThe duration (e.g., x) for identifying the portion 512 associated with the reflector in image 521 can be longer than the interval (e.g., t2-t1) between the time points when images 521, 522, 523, and 524 are acquired. For example, before the time point t1+x when the wearable device 101 identifies the portion associated with the reflector from image 521, images 522 and 523 can be acquired after image 521. Since reflector detection with respect to image 521 is performed until the time point t1+x, reflector detection with respect to images 522 and 523 acquired before the time point t1+x can be delayed. Because the duration for reflector detection is relatively long, the delay can increase as the wearable device 101 acquires images 521, 522, 523, and 524 consecutively.

[0111] According to an embodiment, wearable device 101 can identify regions 552, 553, and 554 associated with a reflector in other images 522, 523, and 524 based on region 551 in image 521 obtained at time point t1+x and / or one or more feature points used to distinguish region 551. Wearable device 101 can assign an identifier (ID) to each of the feature points in each of images 521, 522, 523, and 524. Feature points commonly included in at least two of images 521, 522, 523, and 524 can have the same identifier in both images. For example, feature point f1 commonly included in all images 521, 522, 523, and 524 can have an identifier matched by wearable device 101 in all images 521, 522, 523, and 524. For example, feature point f4 commonly included in all images 522, 523, and 524 can have an identifier matched in all images 522, 523, and 524. For example, the identifier assigned to the feature points included in image 521 may be different from the identifier assigned to feature point f4.

[0112] The wearable device 101, which identifies region 551 in image 521 for distinguishing feature points associated with a reflector, can examine or identify a region in another image corresponding to a reflector by using identifiers of feature points (e.g., feature point f1) on the boundary line of region 551. For example, by searching for feature points in image 522 with identifiers of feature points (e.g., feature point f1) on the boundary line of region 551, the wearable device 101 can extract region 552 in image 522 for distinguishing feature points associated with a reflector. In image 522, feature points f2 and f3, which were not extracted from image 521, are included in region 552 and can therefore be identified as associated with a reflector.

[0113] For example, by searching for feature points in image 523 that correspond to feature points on the boundary line of region 551, wearable device 101 can identify region 553 in image 523 for distinguishing feature points associated with reflectors. Feature points f2, f3, f5, and f6 included in region 553 can be identified as associated with reflectors.

[0114] Similarly, wearable device 101 can identify region 554 for distinguishing feature points associated with a reflector based on feature points in image 524 that match feature points on the boundary line of region 551. For example, in image 524, feature point f4 outside region 554 can be determined not to be associated with a reflector, and feature points f2 and f3 inside region 554 can be determined to be associated with a reflector.

[0115] As described above, by comparing feature points in images 521, 522, 523, and 524 obtained sequentially in the time domain, wearable device 101 can identify or determine regions 552, 553, and 554 in other images 522, 523, and 524 that match the region 551 associated with the reflector in a specific image 521. For example, feature points used to distinguish regions 551, 552, 553, and 554 (e.g., feature points placed on the boundaries of regions 551, 552, 553, and 554, such as feature point f1) may have identifiers that match in each of images 521, 522, 523, and 524. Based on the continuity of the identifiers of the feature points, wearable device 101 can identify variations in the portion 512 associated with the reflector in images 521, 522, 523, and 524.

[0116] Wearable device 101 can calculate the three-dimensional spatial coordinates of each of the feature points extracted from images 521, 522, 523, and 524. Wearable device 101 can calculate the three-dimensional spatial coordinates of each feature point based on whether each feature point is associated with a reflector. For example, in the case where a feature point is associated with a reflector, wearable device 101 can compensate for reprojection errors of the feature point caused by the optical path of reflected light. Embodiments are not limited thereto, and among the feature points in the images (e.g., images 521, 522, 523, and 524), wearable device 101 can preferentially calculate the three-dimensional spatial coordinates of another feature point that is different from the feature point associated with the reflector, or can bypass the calculation of the three-dimensional spatial coordinates of the feature point associated with the reflector.

[0117] As described above, the wearable device 101 according to the embodiment can track the portion 512 associated with the reflector in each of images 521, 522, 523, and 524 obtained from the camera by using feature points in those images. Based on the result of tracking the reflector, the wearable device 101 can calculate its position and / or orientation in external space. Hereinafter, reference will be made to… Figure 6a , Figure 6b and / or Figure 6c Describes the operation of wearable device 101 in calculating position and / or orientation based on feature points.

[0118] Figure 6a , Figure 6b and Figure 6c An example of the operation of the wearable device 101 in identifying the position of the wearable device 101 in an external space including a reflector is shown. Figure 1 , Figure 2a and Figure 2b Wearable device 101 and / or Figure 2a The processor 210 can execute reference Figure 6a , Figure 6b and / or Figure 6c The operation of the wearable device 101 is described.

[0119] Reference Figure 6a , Figure 6b and / or Figure 6c This illustrates different states 601, 602, and 603 distinguished by the movement of the wearable device 101 and / or the user 110 wearing the wearable device 101. (Refer to...) Figure 6a The diagram illustrates a state 601 where user 110 enters a first external space 611 along path 619 and then moves to point p1 outside the first external space 611. The first external space 611 and a second external space 612 adjacent to the first external space 611 may be referred to as a room. In state 601, wearable device 101 can identify feature points fx and / or feature points fy from images and / or videos obtained in the first external space 611. Figure 6a The diamond-shaped marker ◆ can represent the three-dimensional position of feature points extracted from images and / or videos about the first external space 611.

[0120] When worn by user 110, wearable device 101 can be based on Figure 2a and / or Figure 2b The position tracker 271 (or head tracker) performs actions to calculate, track, or monitor the position and / or orientation of the wearable device 101. Figure 6aThe diamond-shaped markers ◆ indicate feature points that can be used to generate map information about the first external space 611. The map information corresponding to the first external space 611 may include feature points and / or keyframes from images and / or videos obtained in the first external space 611. The feature points included in the map information may be included in a point cloud about the first external space 611. The point cloud about the first external space 611 may be provided as a software application executed by the wearable device 101 based on calls to an API corresponding to the location tracker.

[0121] Reference Figure 6a Based on images and / or videos obtained in the first external space 611, the wearable device 101 can determine that there are no reflectors in the first external space 611.

[0122] Reference Figure 6b The image shows a state 602 where the user 110 moves along path 629 from point p1 to point p2 in the second external space 612. Based on the images and / or video obtained in state 602, the wearable device 101 can identify feature points corresponding to the landmarks of the second external space 612. Figure 6b The triangular marker ▲ and / or circular marker ○ can represent the three-dimensional position of feature points extracted from images and / or videos about the second external space 612.

[0123] Reference Figure 6b From images and / or videos obtained in a second external space 612 including reflector 621, wearable device 101 can identify feature points associated with reflected light reflected by reflector 621. (Refer to...) Figure 6b In the reflector 621, when light about feature points f1, f2, f3, and f4 is reflected, the wearable device 101 can identify feature points f1', f2', f3', and f4' respectively corresponding to feature points f1, f2, f3, and f4. (Refer to...) Figure 6b Feature points f1', f2', f3', and f4' based on the reflected light can be determined to be located outside the reflector 621 relative to the wearable device 101. According to an embodiment, the wearable device 101 can assign attributes indicating the association between the feature points associated with the reflected light and / or the reflector 621 to the feature points (e.g., feature points f1', f2', f3', and f4' indicated by the ○ marker). The wearable device 101 can store the feature points obtained in state 602 (e.g., feature points indicated by the ▲ and ○ markers) in map information associated with the second external space 612.

[0124] According to an embodiment, the wearable device 101 can compare feature points obtained in state 602 (e.g., feature points indicated by markers ▲ and ○) with feature points identified based on the first external space 611 (e.g., feature points indicated by marker ◆). For example, the wearable device 101 can compare the remaining feature points (e.g., feature points f1, f2, f3, and f4 indicated by marker ▲) excluding feature points corresponding to reflected light from reflector 621 (e.g., feature points f1', f2', f3', and f4' indicated by marker ○) with feature points identified based on the first external space 611 (e.g., feature points indicated by marker ◆). Based on the comparison, the wearable device 101 can determine whether the user 110 has re-entered the first external space 611 (e.g., loop closed). (Refer to...) Figure 6b Since the feature points extracted from the image and / or video in state 602 (e.g., feature points indicated by the marker ◆) are different from the feature points associated with the first external space 611 (e.g., feature points indicated by the marker ◆), the wearable device 101 can determine that the user 110 has not re-entered the first external space 611.

[0125] As described above, according to the embodiments, the wearable device 101 may use each of the feature points to calculate the position and / or orientation of the wearable device 101 based on the location of the feature points (e.g., the depth of the feature points relative to the wearable device 101), the size of the reflector 621 identified from the image, and / or whether the feature points are associated with the reflector 621.

[0126] Reference Figure 6c This illustrates state 603, where user 110 moves along path 639 from point p2 to point p3 in the first external space 611. In state 603, wearable device 101 can identify user 110's re-entry into the first external space 611 by searching for feature points (e.g., feature points indicated by the marks ◆ and ○) in images and / or videos that have a three-dimensional position corresponding to the first external space 611. For example, wearable device 101 can search among the feature points indicated by the marks ◆ and ○ from a camera (e.g., ...). Figure 2a The remaining feature points in the images and / or videos obtained by the camera 225 and / or the outward-facing camera 225-2, excluding the feature points associated with the reflector 621 (e.g., feature points f1', f2', f3', and f4' indicated by the marker ○).

[0127] For example, if feature points fx and / or fy are identified from an image and / or video obtained in state 603, wearable device 101 can determine that user 110 has re-entered the first external space 611. Upon determining that user 110 has re-entered the first external space 611, wearable device 101 can more accurately calculate the position and / or orientation of wearable device 101 in the first external space 611 based on the positions of feature points fx and / or fy identified in the image and / or video. Wearable device 101 can compensate for errors in the position and / or orientation previously calculated in the first external space 611 based on the calculated position and / or orientation. Error compensation can be performed based on backpropagation.

[0128] For example, feature points f1', f2', f3', and f4' included in the first external space 611 have attributes that indicate their association with reflector 621, so they can be excluded from the calculation of the position and / or orientation of wearable device 101 (e.g., loop closure). Embodiments are not limited thereto. In a first external space 611, different from the second external space 612 that includes reflector 621, feature points associated with reflector 621 can be excluded from the calculation of the position and / or orientation of wearable device 101.

[0129] In the second external space 612 including the reflector 621, the wearable device 101 can calculate its position and / or orientation by using feature points associated with the reflector 621. For example, feature points f1', f2', f3', and f4' indicated as associated with the reflector 621 can be used to more accurately determine the position of the wearable device 101 relative to the reflector 621 in the second external space 612. For example, to reduce errors in calculating the position and / or orientation of the wearable device 101 using remaining feature points (e.g., feature points indicated by the ▲ mark) other than the feature points f1', f2', f3', and f4' associated with the reflector 621, the wearable device 101 can utilize the feature points f1', f2', f3', and f4' associated with the reflector 621.

[0130] In one embodiment, the wearable device 101 can perform a first loop closure after excluding feature points associated with reflector 621 and having three-dimensional positions in the first external space 611 (e.g., feature points f1', f2', f3', and f4' indicated by the marker ○). Based on the first loop closure, the wearable device 101 can search for feature points indicated by markers ◆ and ○ in images and / or videos obtained from the camera. Based on the search, the wearable device 101 can determine whether the user 110 has re-entered the first external space 611.

[0131] The wearable device 101, which performs the first loop closure, can perform a second loop closure based on feature points associated with the reflector 621 and having a three-dimensional position in the first external space 611. Based on the second loop closure, the wearable device 101 can search for feature points indicated by the marks ◆ and ○ in images and / or videos acquired from the camera. Based on the search, the wearable device 101 can determine whether the user 110 has re-entered the first external space 611. Based on the first loop closure and / or the second loop closure, the wearable device 101 can determine the location of the user 110 more accurately. For example, if the location of the user 110 is not determined based on the second loop closure, the wearable device 101 can determine the location of the user 110 based on the first loop closure. The inability to determine the location of the user 110 based on the loop closure may include situations where fewer than a certain number and / or a certain proportion of feature points are found in the images and / or videos.

[0132] In an embodiment, the wearable device 101 may use feature points associated with reflector 621 (e.g., feature points f1', f2', f3', and f4' indicated by the marker ○) to calculate the position and / or orientation of the wearable device 101. The priority may be associated with the number of times the feature points are extracted from images obtained at different time points. For example, as the number of extractions increases, the feature points may have a relatively higher priority. The wearable device 101 may use feature points with relatively high priority to calculate the position and / or orientation of the wearable device 101.

[0133] In an embodiment, the wearable device 101 may calculate its position and / or orientation using feature points associated with the reflector 621 based on the shape and / or reflectivity of the reflector 621. For example, if the reflectivity of the reflector 621 is non-uniform and / or the reflector 621 has a curved shape, the wearable device 101 may not use feature points associated with the reflector 621 to calculate its position and / or orientation. For example, non-uniform reflectivity and / or curved shape of the reflector 621 may cause distortion in the portion of the image obtained by the wearable device 101 corresponding to the reflector 621. The wearable device 101 may discard or ignore feature points or bypass feature point-based calculations, such that feature points extracted in the distorted portion are not used to calculate the position and / or orientation of the wearable device 101. The shape and / or reflectivity of the reflector 621 may be included in the calculation based on... Figure 4 The reflector information 420 is obtained from the neural network 410.

[0134] In an embodiment, the wearable device 101 may use feature points to calculate the position and / or orientation of the wearable device 101 based on the three-dimensional position of feature points associated with the reflector 621. For example, if the depth of the feature point (or the distance of the feature point relative to the reflector 621) is greater than a threshold distance, the wearable device 101 may use other feature points (e.g., feature points not associated with the reflector 621) to calculate the position and / or orientation of the wearable device 101, or it may not use feature points to calculate the position and / or orientation of the wearable device 101.

[0135] As described above, according to the embodiments, wearable device 101 can use images and / or videos acquired at a specific point in time to identify the position and / or orientation of wearable device 101. Wearable device 101 can acquire images and / or videos from a camera to calculate the position and / or orientation of wearable device 101 independently of the gradually increasing errors of sensors (e.g., GPS sensors and / or IMUs) in the time domain. Wearable device 101 can identify at least a portion associated with reflector 621 in the images and / or videos. By using at least a portion, wearable device 101 can reduce the error in the position and / or orientation of wearable device 101 caused by reflector 621.

[0136] As described above, the wearable device 101 according to the embodiment can identify the external space including the wearable device 101 in the first external space 611 and / or the second external space 612 based on feature points of an image obtained using a camera. Based on identifying the position of the wearable device 101 included in the first external space 611 near the second external space 612 including the reflector 621, the wearable device 101 can obtain information associated with the first external space 611 (e.g., the position and / or orientation of the wearable device 101 in the first external space 611) based on another feature point (e.g., the feature point indicated by the markers ◆ and ○) that is different from the feature point associated with the reflector 621 (e.g., the feature point indicated by the marker ○) among the feature points having coordinate values ​​corresponding to the first external space 611.

[0137] Based on identifying the position of the wearable device 101 included in the second external space 612, the wearable device 101 can obtain information associated with the second external space 612 (e.g., the position and / or orientation of the wearable device 101 in the second external space 612) by using feature points with coordinate values ​​corresponding to the second external space 612 (e.g., feature points indicated by the mark ▲). When located in the second external space 612 including the reflector 621, the wearable device 101 can calculate or identify the position and / or orientation of the wearable device 101 in the second external space 612 by using feature points associated with the reflector 621 (e.g., feature points indicated by the mark ○).

[0138] In the following text, reference will be made to Figure 7 The UI displayed by the wearable device 101 that identifies the reflector 621 is described exemplarily.

[0139] Figure 7 An example of a user interface (UI) displayed by a wearable device 101 that identifies the reflector 130 is shown. Figure 1 , Figure 2a and Figure 2b Wearable device 101 and / or Figure 2a The processor 210 can execute reference Figure 7 The operation of the wearable device 101 is described.

[0140] According to an embodiment, the wearable device 101 may display a camera (e.g., a sensor) Figure 2a The screen 710 displays images and / or videos obtained by the camera 225 and / or the outward-facing camera 225-2. The screen 710 may be displayed on the display of the wearable device 101 based on the execution of software applications according to AR, MR, and / or VST. Figure 2a On display 220. In screen 710, wearable device 101 can display virtual object 720 together with visual objects 141, 142, 143, and 144 corresponding to external objects. Virtual object 720 may have a three-dimensional position mapped to external space visible through screen 710. For example, wearable device 101 may provide a visual effect of floating in external space based on the movement and / or rotation of virtual object 720 in screen 710.

[0141] Wearable device 101 can identify reflectors 130 adjacent to it based on images and / or videos acquired from a camera. The wearable device 101, having identified portions of the image associated with the reflector 130, can display a visual object on its display for verifying the reflector 130. Figure 7In the exemplary screen 710, the wearable device 101 can highlight the portion 732 corresponding to the reflector 130 by using a visual object having lines and / or graphic shapes. The wearable device 101 may display a virtual object 730 for notifying the recognition of the reflector 130. The virtual object 730 may include preset text (e.g., “reflector detection”) indicating the recognition of the reflector 130.

[0142] In an embodiment, when recognizing or identifying the reflector 130 in an image, the wearable device 101 may determine whether to display the virtual object 730 based on the number of feature points in a second portion of the image that differs from the first portion corresponding to the reflector 130 (e.g., a portion adjacent to the first portion). For example, if more than a preset number of feature points are extracted in the second portion, the wearable device 101 may not display the virtual object 730. In this example, if less than a preset number of feature points are extracted in the second portion, the wearable device 101 may display the virtual object 730. For example, if less than a preset number of feature points are extracted in the second portion during a preset duration, the wearable device 101 may display the virtual object 730. In this example, if more than a preset number of feature points are extracted in the second portion, the wearable device 101 may not display the virtual object 730.

[0143] The wearable device 101, which identifies the reflector 130, can perform operations to obtain additional information associated with the reflector 130. For example, the wearable device 101 can guide the acquisition of images and / or videos of the external space adjacent to the reflector 130 based on a virtual object 730. For example, the wearable device 101 can guide the user 110 wearing the wearable device 101 to a location adjacent to the reflector 130. Based on this operation, the wearable device 101 can additionally obtain feature points associated with the reflector 130 and / or the external space adjacent to the reflector 130. The feature points additionally obtained by the wearable device 101 can be used to more accurately generate map information about the external space including the reflector 130.

[0144] In an exemplary state where reflector 130 is identified, wearable device 101 can provide a user experience based on reflector 130. For example, wearable device 101 can provide functionality associated with virtual object 720 by using visual object 143 on screen 710 corresponding to reflector 130 and / or portion 732. For example, wearable device 101 can provide visual effects based on the distance between visual object 143 and virtual object 720, such as virtual object 720 being attached to visual object 143. When virtual object 720 and visual object 143 are connected to each other, wearable device 101 can move virtual object 720 on visual object 143 based on a virtual coefficient of friction associated with reflector 130. For example, virtual object 720 can slide on visual object 143 at a relatively high speed. Embodiments are not limited thereto, and wearable device 101 can perform the function of replicating virtual object 720 based on contact between visual object 143 and virtual object 720.

[0145] For example, wearable device 101 may display a mirror image corresponding to virtual object 720 in a portion 732 of screen 710 associated with reflector 130. Based on the mirror image displayed in portion 732, wearable device 101 can provide visual effects such as virtual object 720 being reflected on reflector 130.

[0146] As described above, the wearable device 101 according to the embodiment can identify a reflector 130 in external space. Based on the result of identifying the reflector 130, the wearable device 101 can prevent degradation of position tracking (e.g., head tracking) performance caused by the reflector 130. For example, the wearable device 101 can improve position tracking performance by using feature points associated with the reflected light from the reflector 130. This is achieved by considering the computational cost required to identify the portion 732 corresponding to the reflector 130 (e.g., for operation). Figure 4 (The computational cost of the neural network 410) The wearable device 101 can use a portion 732 obtained from an image at a specific time point to identify the portion 732 corresponding to the reflector 130 from images at different time points.

[0147] In the following text, reference will be made to Figure 8 Exemplary operation of wearable device 101 according to an embodiment is described.

[0148] Figure 8 An example flowchart of a wearable device according to an embodiment is shown. Figure 1 , Figure 2a and Figure 2b Wearable device 101 and / or Figure 2a The processor 210 can execute reference Figure 8 The described operation. Figure 8 At least one of the operations can be performed with Figure 2aand / or Figure 2b The location tracker 271 is associated with the operation of the wearable device. Figure 8 Operation can be combined with Figure 3 The operations are related.

[0149] In the following embodiments, each operation may be executed sequentially, but not necessarily in a sequential order. For example, the order of each operation may be changed, and at least two operations may be executed in parallel.

[0150] Reference Figure 8 In operation 810, the processor of the wearable device according to the embodiment can use a camera (e.g., Figure 2a The camera 225 and / or the outward-facing camera 225-2) acquire images of the external space (e.g., Figure 1 Images 140 and 150 and / or Figure 5a Images 521, 522, 523 and 524).

[0151] Reference Figure 8 In operation 820, the processor of the wearable device according to the embodiment can identify whether at least a portion of the image from operation 810 is associated with a reflector of reflected light (e.g., Figure 1 Reflector 130 Figure 5a The reflector 510 and / or Figures 6a to 6c Associated with reflector 621). Based on Figure 2b and / or Figure 4 The execution of the reflector detector 295 allows the processor to identify portions of the image associated with reflectors (e.g., Figure 4 Part 430 Figures 5a to 5b Part 512 and / or Figure 7 (Part 732). In a state where at least a portion of the image associated with the reflector in operation 820 is identified (820-Yes), the processor of the wearable device can execute operation 830. In a state where at least a portion of the image associated with the reflector in operation 820 is not identified (820-No), the processor of the wearable device can execute operation 840.

[0152] Reference Figure 8 In operation 830, the processor of the wearable device according to the embodiment can obtain information indicating whether each of the feature points in the image is included in at least a portion of the image associated with the reflector. The information in operation 830 may include attributes of each of the feature points. The information may include logical values ​​(e.g., Boolean data types), flags, and / or probability values ​​indicating whether the feature point corresponds to the reflector.

[0153] Reference Figure 8In operation 840, the processor of the wearable device according to the embodiment can identify external space based on feature points of an image. The processor can identify the shape of the external space and / or one or more external objects included in the external space based on the three-dimensional position of the feature points. The processor can generate or obtain information indicating the external space based on operation 840. The result of identifying the external space may include map information (and / or a local map) corresponding to the external space.

[0154] Reference Figure 8 In operation 850, the processor of the wearable device according to an embodiment can provide a virtual space at least partially mapped to the external space based on the result of identifying the external space. The processor can execute software applications installed in the wearable device (e.g., including...). Figure 2a Instructions for providing a virtual space in the software application (application layer 240). By executing instructions based on the result of identifying the external space in operation 840, the processor can obtain a virtual space that is at least partially mapped to the external space. The processor can then display (e.g., on a monitor). Figure 2a At least a portion of the obtained virtual space is displayed on the monitor 220.

[0155] In the following text, refer to Figure 9a , Figure 9b , Figure 10a and / or Figure 10b Show reference Figures 1 to 8 An exemplary exterior of the wearable device described. Figure 9a and / or Figure 9b Wearable devices 900 and / or Figure 10a and / or Figure 10b The wearable device 1000 can be Figure 1 Example of wearable device 101.

[0156] Figure 9a An example perspective view of a wearable device according to an embodiment is shown. According to an embodiment, the wearable device 900 may be in the form of glasses that can be worn on a part of a user's body (e.g., the head). The wearable device 900 may include a head-mounted display (HMD). For example, the housing of the wearable device 900 may be in the form of a flexible material (such as rubber and / or silicone) having portions that are tightly attached to the user's head (e.g., portions of the face surrounding the two eyes). For example, the housing of the wearable device 900 may include one or more straps that can be wrapped around the user's head, and / or one or more temples that can be attached to the ears of the head.

[0157] Reference Figure 9a According to an embodiment, the wearable device 900 may include at least one display 950 and a frame 909 supporting the at least one display 950.

[0158] According to an embodiment, the wearable device 900 can be worn on a part of a user's body. The wearable device 900 can provide the user wearing the wearable device 900 with augmented reality (AR), virtual reality (VR), or a combination of augmented reality and virtual reality (MR). For example, the wearable device 900 can respond to... Figure 9b The user's preset poses are obtained from motion recognition cameras 960-2 and 960-3 and displayed on at least one display 950. Figure 9b Virtual reality images provided by at least one optical device 982 and 984.

[0159] According to an embodiment, at least one display 950 may provide visual information to a user. For example, at least one display 950 may include a transparent or translucent lens. At least one display 950 may include a first display 950-1 and / or a second display 950-2 spaced apart from the first display 950-1. For example, the first display 950-1 and the second display 950-2 may be positioned corresponding to the user's left and right eyes, respectively.

[0160] Reference Figure 9b At least one display 950 can provide a user with visual information from ambient light transmitted through a lens included in at least one display 950, as well as other visual information different from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display 950 may include a first surface 931 and a second surface 932 opposite to the first surface 931. A display area can be formed on the second surface 932 of at least one display 950. When a user wears a wearable device 900, ambient light can be transmitted to the user by incident on the first surface 931 and passing through the second surface 932. As another example, at least one display 950 can display an augmented reality image on a display area formed on the second surface 932, in which a virtual reality image provided by at least one optics 982 and 984 is combined with a real screen transmitted through ambient light.

[0161] In an embodiment, at least one display 950 may include at least one waveguide 933 and 934 that transmits light emitted from at least one optical device 982 and 984 to a user via diffraction. At least one waveguide 933 and 934 may be formed based on at least one of glass, plastic, or polymer. Nanopatterns may be formed on at least a portion of the exterior or interior of at least one waveguide 933 and 934. The nanopatterns may be formed based on a grating structure having a polygonal or curved shape. Light incident on one end of at least one waveguide 933 and 934 may be propagated through the nanopattern to the other end of at least one waveguide 933 and 934. At least one waveguide 933 and 934 may include at least one of at least one diffractive element (e.g., a diffractive optics element (DOE), a holographic optics element (HOE)) and a reflective element (e.g., a mirror). For example, at least one waveguide 933 and 934 may be disposed in a wearable device 900 to guide a screen displayed by at least one display 950 to the user's eyes. For example, the screen can be transmitted to the user's eye based on total internal reflection (TIR) ​​generated in at least one waveguide 933 and 934.

[0162] Wearable device 900 can analyze objects included in real images collected by imaging cameras 960-4, combine them with virtual objects, and display them on at least one display 950, wherein the virtual objects correspond to objects among the analyzed objects that become subjects provided by augmented reality. The virtual objects may include at least one of text and images for associating with objects included in the real images. Wearable device 900 can analyze objects based on multiple cameras (such as stereo cameras). For object analysis, wearable device 900 can use multiple cameras and / or Time-of-Flight (ToF) to perform spatial recognition (e.g., Simultaneous Localization and Mapping (SLAM)). A user wearing wearable device 900 can view the images displayed on at least one display 950.

[0163] According to an embodiment, the frame 909 may be configured with a physical structure that allows the wearable device 900 to be worn on a user's body. According to an embodiment, the frame 909 may be configured such that when the user wears the wearable device 900, the first display 950-1 and the second display 950-2 may be positioned corresponding to the user's left and right eyes. The frame 909 may support at least one display 950. For example, the frame 909 may support the first display 950-1 and the second display 950-2 positioned corresponding to the user's left and right eyes.

[0164] Reference Figure 9aAccording to an embodiment, the frame 909 may include a region 920 that at least partially contacts a portion of the user's body when the user is wearing the wearable device 900. For example, the region 920 of the frame 909 that contacts a portion of the user's body may include areas that contact the user's nose, ears, and sides of the user's face that are in contact with the wearable device 900. According to an embodiment, the frame 909 may include a nose pad 910 that contacts a portion of the user's body. When the wearable device 900 is worn by the user, the nose pad 910 may contact the portion of the user's nose. The frame 909 may include a first temple 904 and a second temple 905 that contact another portion of the user's body, different from that portion of the user's body.

[0165] For example, frame 909 may include a first bezel 901 surrounding at least a portion of a first display 950-1, a second bezel 902 surrounding at least a portion of a second display 950-2, a bridge 903 disposed between the first bezel 901 and the second bezel 902, a first pad 911 disposed along the edge of the first bezel 901 from one end of the bridge 903, a second pad 912 disposed along the edge of the second bezel 902 from the other end of the bridge 903, a first temple 904 extending from the first bezel 901 and secured to the wearer's ear, and a second temple 905 extending from the second bezel 902 and secured to the ear opposite the ear. The first pad 911 and the second pad 912 may contact the user's nose, and the first temple 904 and the second temple 905 may contact the user's face and the user's ear. Temple 904 or 905 can be... Figure 9b Hinges 906 and 907 are rotatably connected to the frame. A first temple 904 is rotatably connected relative to the first frame 901 via a first hinge unit 906 disposed between the first frame 901 and the first temple 904. A second temple 905 is rotatably connected relative to the second frame 902 via a second hinge unit 907 disposed between the second frame 902 and the second temple 905. According to an embodiment, the wearable device 900 can identify external objects touching the frame 909 (e.g., a user's fingertips) and / or gestures performed by external objects by using touch sensors, grip sensors, and / or proximity sensors formed on at least a portion of the surface of the frame 909.

[0166] According to an embodiment, the wearable device 900 may include hardware that performs various functions (e.g., based on the above). Figure 2a , Figure 2b and / or Figure 4The block diagram describes the hardware. For example, the hardware may include a battery module 970, an antenna module 975, at least one optics device 982 and 984, a speaker (e.g., speaker 955-1 and 955-2), a microphone (e.g., microphone 965-1, 965-2 and 965-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) 990 (e.g., a printed circuit board). Various hardware components may be disposed in frame 909.

[0167] According to an embodiment, the microphones of the wearable device 900 (e.g., microphones 965-1, 965-2, and 965-3) can acquire sound signals by being disposed on at least a portion of the frame 909. Figure 9b The image shows a first microphone 965-1 mounted on the bridge of the nose 903, a second microphone 965-2 mounted on the second frame 902, and a third microphone 965-3 mounted on the first frame 901. However, the number and arrangement of the microphones 965 are not limited to these specifications. Figure 9b In an embodiment where the number of microphones 965 included in the wearable device 900 is two or more, the wearable device 900 can identify the direction of the sound signal by using multiple microphones disposed on different parts of the frame 909.

[0168] According to an embodiment, at least one optical device 982 and 984 can project virtual objects onto at least one display 950 to provide various image information to a user. For example, at least one optical device 982 and 984 can be a projector. At least one optical device 982 and 984 can be configured to be adjacent to at least one display 950, or can be included as part of at least one display 950. According to an embodiment, a wearable device 900 may include a first optical device 982 corresponding to a first display 950-1 and a second optical device 984 corresponding to a second display 950-2. For example, at least one optical device 982 and 984 may include a first optical device 982 disposed on the periphery of the first display 950-1 and a second optical device 984 disposed on the periphery of the second display 950-2. The first optical device 982 can emit light to a first waveguide 933 disposed on the first display 950-1, and the second optical device 984 can emit light to a second waveguide 934 disposed on the second display 950-2.

[0169] In an embodiment, camera 960 may include a capturing camera 960-4, an eye-tracking camera (ET CAM) 960-1, and / or motion recognition cameras 960-2 and 960-3. The capturing camera 960-4, the eye-tracking camera 960-1, and the motion recognition cameras 960-2 and 960-3 may be positioned at different locations on the frame 909 and may perform different functions. The eye-tracking camera 960-1 may output data indicating the position or gaze of the eyes of a user wearing the wearable device 900. For example, the wearable device 900 may detect a gaze from an image including the user's pupils obtained through the eye-tracking camera 960-1.

[0170] Wearable device 900 can identify objects (e.g., real and / or virtual objects) focused by the user using the user's gaze obtained through eye-tracking camera 960-1. Wearable device 900, which identifies the focused object, can perform functions for interaction between the user and the focused object (e.g., gaze interaction). Wearable device 900 can represent in virtual space the portion corresponding to the eyes of the user's avatar using the user's gaze obtained through eye-tracking camera 960-1. Wearable device 900 can render an image (or screen) displayed on at least one display 950 based on the position of the user's eyes.

[0171] For example, the visual quality (e.g., resolution, brightness, saturation, grayscale, and PPI) of a first region within an image that is gaze-dependent may differ from the visual quality of a second region distinct from the first region. Wearable device 900 can obtain an image with the visual quality of the first and second regions matching the user's gaze by using foveated rendering. For example, when wearable device 900 supports iris recognition, user authentication can be performed based on iris information obtained using eye-tracking camera 960-1. Figure 9b An example of setting up an eye-tracking camera 960-1 toward the user's right eye is shown, but the embodiments are not limited thereto, and the eye-tracking camera 960-1 may be set up toward the user's left eye alone or toward both eyes.

[0172] In an embodiment, the capturing camera 960-4 can capture a real image or background to match with a virtual image to achieve augmented reality or mixed reality content. The capturing camera 960-4 can be used to obtain images with high resolution based on high resolution (HR) or photo-video (PV). The capturing camera 960-4 can capture an image of a specific object present at a location viewed by a user and can provide the image to at least one display 950. At least one display 950 can display an image in which a virtual image provided by at least one optics device 982 and 984 is overlaid with information about a real image or background including an image of a specific object obtained by using the capturing camera 960-4. The wearable device 900 can compensate for depth information (e.g., the distance between the wearable device 900 and an external object obtained by a depth sensor) by using the image obtained by the capturing camera 960-4. The wearable device 900 can perform object recognition by using the image obtained by the capturing camera 960-4. The wearable device 900 can perform functions such as focusing on objects (or subjects) within an image (e.g., autofocus) and / or optical image stabilization (OIS) (e.g., image stabilization) using a camera 960-4. When a screen representing a virtual space is displayed on at least one display 950, the wearable device 900 can perform a pass-through function to display an image obtained by the camera 960-4 overlapping at least a portion of the screen. In an embodiment, the camera 960-4 may be mounted on a nose bridge 903, which is positioned between a first bezel 901 and a second bezel 902.

[0173] An eye-tracking camera 960-1 can achieve more realistic augmented reality by tracking the gaze of a user wearing a wearable device 900 and matching the user's gaze with visual information provided on at least one display 950. For example, when the user is looking forward, the wearable device 900 can naturally display environmental information associated with the area in front of the user on at least one display 950 at the user's location. The eye-tracking camera 960-1 can be configured to capture images of the user's pupils to determine the user's gaze. For example, the eye-tracking camera 960-1 can receive gaze detection light reflected from the user's pupils and can track the user's gaze based on the position and movement of the received gaze detection light. In an embodiment, the eye-tracking camera 960-1 can be positioned corresponding to the user's left and right eyes. For example, the eye-tracking camera 960-1 can be positioned in a first bezel 901 and / or a second bezel 902 to face the direction in which the user wearing the wearable device 900 is located.

[0174] Motion recognition cameras 960-2 and 960-3 can provide specific events to a screen provided on at least one display 950 by recognizing movement of the whole or parts of a user's body (such as the user's torso, hands, or face). Motion recognition cameras 960-2 and 960-3 can obtain signals corresponding to user movements (e.g., gesture recognition) and can provide a display corresponding to the signals to at least one display 950. A processor can recognize signals corresponding to operations and can perform preset functions based on this recognition. Motion recognition cameras 960-2 and 960-3 can be used to perform simultaneous localization and mapping (SLAM) and / or spatial recognition functions using depth maps in 6-DOF poses. A processor can perform pose recognition and / or object tracking functions using motion recognition cameras 960-2 and 960-3. In an embodiment, motion recognition cameras 960-2 and 960-3 can be mounted on a first frame 901 and / or a second frame 902.

[0175] The camera 960 included in the wearable device 900 is not limited to the eye-tracking camera 960-1 and motion recognition cameras 960-2 and 960-3 described above. For example, the wearable device 900 can identify external objects included in the FoV by using a camera positioned facing the user's FoV. The identification of external objects by the wearable device 900 can be performed based on sensors (such as depth sensors and / or time-of-flight (ToF) sensors) used to identify the distance between the wearable device 900 and the external object. The camera 960 facing the FoV may support autofocus and / or optical image stabilization (OIS) functions. For example, to obtain an image including the face of a user wearing the wearable device 900, the wearable device 900 may include a camera 960 facing the face (e.g., a face-tracking (FT) camera).

[0176] Although not shown, the wearable device 900 according to an embodiment may also include a light source (e.g., an LED) that emits light toward an object captured by the camera 960 (e.g., the user's eyes, face, and / or an external object in FoV). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame 909 and hinge units 906 and 907.

[0177] According to an embodiment, the battery module 970 can power the electronic components of the wearable device 900. In an embodiment, the battery module 970 may be disposed in the first temple 904 and / or the second temple 905. For example, there may be multiple battery modules 970. The multiple battery modules 970 may be disposed on each of the first temple 904 and the second temple 905 respectively. In an embodiment, the battery module 970 may be disposed at the end of the first temple 904 and / or the second temple 905.

[0178] The antenna module 975 can transmit signals or power to the outside of the wearable device 900, or can receive signals or power from the outside. In an embodiment, the antenna module 975 may be disposed in the first temple 904 and / or the second temple 905. For example, the antenna module 975 may be disposed near a surface of the first temple 904 and / or the second temple 905.

[0179] The speaker 955 can output sound signals to the outside of the wearable device 900. The sound output module may be referred to as a speaker. In an embodiment, the speaker 955 may be disposed in the first temple 904 and / or the second temple 905 so as to be positioned adjacent to the ear of the user wearing the wearable device 900. For example, the speaker 955 may include a second speaker 955-2 positioned adjacent to the user's left ear by being disposed in the first temple 904 and a first speaker 955-1 positioned adjacent to the user's right ear by being disposed in the second temple 905.

[0180] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state, or may emit light through operation corresponding to a specific state, in order to visually provide the user with information about a specific state of the wearable device 900. For example, when the wearable device 900 needs charging, it may emit red light at a constant period. In an embodiment, the light-emitting module may be disposed on a first frame 901 and / or a second frame 902.

[0181] Reference Figure 9b According to an embodiment, the wearable device 900 may include a printed circuit board (PCB) 990. The PCB 990 may be included in at least one of a first temple 904 or a second temple 905. The PCB 990 may include an interposer layer disposed between at least two sub-PCBs. On the PCB 990, one or more hardware components included in the wearable device 900 (e.g., by...) may be disposed. Figure 4 (The different blocks shown represent the hardware). Wearable device 900 may include a flexible PCB (FPCB) for interconnecting the hardware.

[0182] According to an embodiment, the wearable device 900 may include at least one of a gyroscope sensor, a gravity sensor, and / or an accelerometer sensor for detecting the posture of the wearable device 900 and / or the posture of a body part (e.g., head) of a user wearing the wearable device 900. Each of the gravity sensor and the accelerometer sensor may measure gravitational acceleration and / or acceleration based on preset three-dimensional axes (e.g., x-axis, y-axis, and z-axis) perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the preset three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, accelerometer sensor, and gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to an embodiment, the wearable device 900 may use the IMU to identify the movements and / or gestures of a user being executed to perform or stop a specific function of the wearable device 900.

[0183] Figures 10a to 10b An example of the exterior of a wearable device according to an embodiment is shown. Figures 10a to 10b The wearable device 1000 may include reference Figure 9a and / or Figure 9b At least a portion of the hardware of the described wearable device 900. According to an embodiment, it is possible to... Figure 10a An example of the exterior of the first surface 1010 of the housing of the wearable device 1000 is shown, and it can be seen that... Figure 10b An example of the exterior of the second surface 1020, which is opposite to the first surface 1010, is shown.

[0184] Reference Figure 10a According to an embodiment, the first surface 1010 of the wearable device 1000 may have a shape that is attachable to a user's body part (e.g., the user's face). Although not shown, the wearable device 1000 may also include a strap and / or one or more temples for securing to the user's body part (e.g., [missing information]). Figures 9a to 9b The first temple 904 and / or the second temple 905. A first display 950-1 for outputting an image to the left eye of the user and a second display 950-2 for outputting an image to the right eye of the user may be provided on the first surface 1010. The wearable device 1000 may also include rubber or silicone packaging formed on the first surface 1010 to prevent interference from light different from the light emitted from the first display 950-1 and the second display 950-2 (e.g., ambient light).

[0185] According to an embodiment, the wearable device 1000 may include a camera 960-1 for capturing and / or tracking the two eyes of a user adjacent to each of the first display 950-1 and the second display 950-2. The camera 960-1 may be referred to as... Figure 9bThe wearable device 1000 may include gaze-tracking cameras 960-1 and 960-6 for capturing and / or recognizing a user's face, according to an embodiment. Cameras 960-5 and 960-6 may be referred to as FT cameras. The wearable device 1000 may control an avatar representing the user in a virtual space based on the movement of the user's face recognized using cameras 960-5 and 960-6. For example, the wearable device 1000 may alter the texture and / or shape of portions of the avatar (e.g., portions representing a human face) by using information obtained by cameras 960-5 and 960-6 (e.g., FT cameras) that represents the facial expressions of the user wearing the wearable device 1000.

[0186] Reference Figure 10b Cameras (e.g., cameras 960-7, 960-8, 960-9, 960-10, 960-11, and 960-12) and / or sensors (e.g., depth sensor 1030) used to acquire information related to the external environment of the wearable device 1000 may be positioned in conjunction with... Figure 10a The first surface 1010 is on the opposite second surface 1020. For example, cameras 960-7, 960-8, 960-9 and 960-10 may be disposed on the second surface 1020 in order to identify external objects. Figure 10b The 960-7, 960-8, 960-9 and 960-10 cameras are compatible with... Figure 9b The motion recognition cameras 960-2 and 960-3 correspond to this.

[0187] For example, by using cameras 960-11 and 960-12, the wearable device 1000 can acquire images and / or videos to be sent to each of the user's two eyes. Camera 960-11 can be disposed on the second surface 1020 of the wearable device 1000 to acquire an image to be displayed via a second display 950-2 corresponding to the right eye. Camera 960-12 can be disposed on the second surface 1020 of the wearable device 1000 to acquire an image to be displayed via a first display 950-1 corresponding to the left eye. Cameras 960-11 and 960-12 can be used with... Figure 9b It is compatible with the 960-4 camera.

[0188] According to an embodiment, the wearable device 1000 may include a depth sensor 1030 disposed on a second surface 1020 to identify the distance between the wearable device 1000 and an external object. By using the depth sensor 1030, the wearable device 1000 can obtain spatial information (e.g., a depth map) about at least a portion of the FoV of the user wearing the wearable device 1000. Although not shown, a microphone for obtaining sound output from an external object may be disposed on the second surface 1020 of the wearable device 1000. According to an embodiment, the number of microphones may be one or more.

[0189] In embodiments, a method may be needed to identify the position and / or orientation of the wearable device in an external space including a reflector. As described above, according to embodiments, the wearable device (e.g., Figure 1 Wearable device 101 Figure 9a and Figure 9b Wearable devices 900 and / or Figure 10a and Figure 10b The wearable device 1000 may include a camera (e.g., Figure 2a The camera 225), and a memory including one or more storage media for storing instructions (e.g., Figure 2a The memory 215) and at least one processor (e.g., Figure 2a The processor 210). When executed individually or jointly by at least one processor, the instructions enable the wearable device to acquire an image of the external space including the wearable device using a camera (e.g., Figure 1 Images 140 and 150 and / or Figure 5a Images 521, 522, 523, and 524). When executed individually or jointly by at least one processor, the instructions enable the wearable device to identify reflective objects in the image (e.g., objects reflecting light) based on object recognition of the image. Figure 1 Reflector 130 Figure 5a The reflector 510 and / or Figures 6a to 6c The part associated with the reflector 621 (e.g., Figure 4 Part 430 Figures 5a to 5b Part 512 and / or Figure 7 (Part 732). When executed individually or jointly by the at least one processor, the instructions enable the wearable device to perform a software application by using information about whether each of the image-based feature points is included in the part, providing a virtual space that is at least partially mapped to external space.

[0190] For example, when executed individually or jointly by at least one processor, the instructions may enable the wearable device to determine, based on the size of the portion in the image, whether at least one feature point identified from that portion is used to identify the wearable device's location in external space.

[0191] For example, when executed individually or jointly by at least one processor, the instructions may cause the wearable device to display a visual object for verifying the reflector on the wearable device's display in response to the portion associated with the reflector.

[0192] For example, when executed individually or jointly by at least one processor, the instructions may enable the wearable device to identify a first feature point surrounding the portion in a feature point of an image acquired at a first time point. When executed individually or jointly by at least one processor, the instructions may enable the wearable device to identify another portion associated with the reflector in another image based on a second feature point that matches the first feature point and is included in another image acquired at a different time point from the first time point.

[0193] For example, when the instructions are executed individually or jointly by at least one processor, the wearable device may identify a second feature point that matches the first feature point based on identifiers assigned to feature points included in another image.

[0194] For example, when executed individually or jointly by at least one processor, the instructions can enable a wearable device to display at least a portion of a virtual space associated with the external space on the wearable device's display by means of instructions for executing a software application that recreates the virtual space based on information.

[0195] For example, when the instructions are executed individually or jointly by at least one processor, the wearable device can determine its location in virtual space based on the location of the wearable device in external space indicated by information.

[0196] For example, when the instructions are executed individually or jointly by at least one processor, the wearable device may identify the location of the portion in an image by using a neural network on which the image is input.

[0197] As described above, according to embodiments, the method of the wearable device may include obtaining an image of the external space including the wearable device using the camera of the wearable device (e.g., Figure 8 Operation 810). The method may include identifying portions of an image associated with reflectors of reflected light based on object recognition of the image. The method may include providing a virtual space at least partially mapped to external space (e.g., by using information based on whether each feature point in the image is included in that portion) to perform a software application. Figure 8 Operation 850).

[0198] For example, the method may include determining, based on the size of the portion in the image, whether at least one feature point identified from that portion is used to identify the location of the wearable device in external space. The method may also include obtaining information indicating external space based on whether at least one feature point is used to identify that location.

[0199] For example, identification may include displaying a visual object on the wearable device's display to verify the reflector in response to the portion associated with the reflector.

[0200] For example, identification may include identifying a first feature point surrounding the portion among feature points in an image obtained at a first time point. The method may also include identifying another portion of the reflector in another image based on a second feature point that matches the first feature point and is included in another image obtained at a different time point than the first time point.

[0201] For example, identifying the other part may include: identifying a second feature point that matches the first feature point based on identifiers assigned to feature points included in the other image.

[0202] For example, it may provide instructions that may include executing a software application for reproducing a virtual space based on information, displaying at least a portion of a virtual space associated with the external space on a display of a wearable device.

[0203] For example, it may provide a method to determine the location of a wearable device in a virtual space based on the location of the wearable device in external space indicated by information.

[0204] For example, recognition may include identifying the location of that part in an image by using a neural network into which the image is input.

[0205] As described above, according to embodiments, wearable devices (e.g., Figure 1 Wearable device 101 Figure 9a and Figure 9b Wearable devices 900 and / or Figure 10a and Figure 10b The wearable device 1000 may include a camera (e.g., Figure 2a The camera 225), and the memory for storing instructions (e.g., Figure 2a The memory 215) and the processor for executing instructions (e.g., Figure 2a The processor 210 can be configured to base its operation on images obtained using a camera (e.g., ...). Figure 1 Images 140 and 150 and / or Figure 5aThe processor can identify the external space including the wearable device based on a first feature point in images 521, 522, 523, and 524. Figures 6a to 6c The first external space 611) or includes a reflector (e.g., Figure 1 Reflector 130 Figure 5a The reflector 510 and / or Figures 6a to 6c The second external space of the reflector 621 (e.g., Figures 6a to 6c In a second external space (612), a wearable device included in a first external space adjacent to the second external space is identified, and a second feature point having coordinate values ​​corresponding to the first external space and identified based on a camera is identified. The processor can be configured to obtain information associated with the first external space based on a fourth feature point among the first and second feature points, the fourth feature point being different from one or more third feature points associated with a reflector. The processor can be configured to: based on the identification of a wearable device included in the second external space in either the first or second external space, obtain information associated with the second external space by using the first feature point and one or more third feature points.

[0206] For example, the processor may be configured to display at least a portion of a virtual space on a display of a wearable device based on the information obtained, wherein at least a portion of the virtual space is mapped to an external space including the wearable device in a first external space or a second external space.

[0207] For example, a processor can be configured to identify a first feature point in an image by using a neural network into which the image is input.

[0208] For example, the processor can be configured to identify one or more third feature points among the second feature points based on the attributes of the second feature points.

[0209] As described above, according to embodiments, a method for using a wearable device may include identifying an external space including the wearable device based on a first feature point in an image obtained using a camera of the wearable device. The method may include identifying a wearable device included in a first external space or a second external space including a reflector, located near the second external space, and identifying a second feature point having coordinate values ​​corresponding to the first external space and identified based on the camera. The method may include obtaining information associated with the first external space based on a fourth feature point, one of the first and second feature points, wherein the fourth feature point is different from one or more third feature points associated with the reflector. The method may include obtaining information associated with the second external space based on identifying a wearable device included in a second external space, using the first feature point and one or more third feature points.

[0210] For example, the method may include: displaying at least a portion of a virtual space on a display of a wearable device based on the information obtained, wherein at least a portion of the virtual space is mapped to an external space including the wearable device in a first external space or a second external space.

[0211] For example, identifying external space may include using a neural network that inputs an image to identify first feature points in the image.

[0212] For example, obtaining information associated with the first external space may include identifying one or more third feature points among the second feature points based on the attributes of the second feature points.

[0213] The aforementioned apparatus can be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the apparatus and components described in the embodiments can be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field-programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or any other means capable of executing and responding to instructions. The processing apparatus can execute an operating system (OS) and one or more software applications executed on the operating system. Furthermore, the processing apparatus can access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, there are cases where a single processing apparatus is described; however, those skilled in the art will recognize that a processing apparatus can include multiple processing elements and / or various types of processing elements. For example, a processing apparatus can include multiple processors or one processor and one controller. Additionally, another processing configuration, such as a parallel processor, is also possible.

[0214] Software may include computer programs, code, instructions, or combinations thereof, and may be configured to operate a processing device as needed or to independently or jointly command the processing device. Software and / or data may be implemented in any type of machine, component, physical device, computer storage medium, or apparatus to be interpreted by the processing device or to provide commands or data to the processing device. Software may be distributed across network-connected computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0215] The method according to the embodiments can be implemented in the form of program commands, which can be executed by various computer means and recorded on a computer-readable medium. In this case, the medium can continuously store a computer-executable program, or it can temporarily store the program for execution or download. Furthermore, the medium can be various recording or storage devices in the form of a single piece of hardware or a combination of several pieces, but is not limited to media directly connected to a computer system, and can exist distributed across a network. Examples of media may include magnetic media (such as hard disks, floppy disks, and magnetic tapes), optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as optical-magnetic floppy disks), and media configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, other examples of media may include recording or storage media managed by application stores that distribute applications, sites that provide or distribute various software, servers, etc.

[0216] Although embodiments have been described above with reference to limited examples and figures, various modifications and variations can be made by those skilled in the art based on the above description. For example, suitable results may be achieved even if the described techniques are performed in a different order than the described methods, and / or components of the described systems, structures, devices, circuits, etc. are coupled or combined in a different form than the described methods, or are replaced or substituted by other components or equivalents.

[0217] Therefore, other implementations, other embodiments, and those equivalent to the scope of the claims are within the scope of the following claims.

Claims

1. A wearable device (101; 900; 1000), comprising: Camera (225); The memory (215) includes one or more storage media that store instructions; and At least one processor (210) includes processing circuitry, wherein the instructions, when executed individually or jointly by the at least one processor, cause the wearable device to perform the following operations: Images of the external space including the wearable device were obtained using a camera (140, 150; 521, 522, 523, 524). Based on object recognition of the image, the portion of the image associated with the reflector (130; 510; 621) reflecting the reflected light (430; 512; 732) is identified. as well as By using information based on whether each feature point in the image is included in the portion, a software application is performed, providing a virtual space that is at least partially mapped to external space.

2. The wearable device as claimed in claim 1, wherein, When executed individually or jointly by the at least one processor, the instructions cause the wearable device to perform the following operations: Based on the size of the portion in the image, determine whether at least one feature point identified from the portion is used to identify the location of the wearable device in external space.

3. The wearable device according to claims 1 to 2, wherein, When executed individually or jointly by the at least one processor, the instructions cause the wearable device to perform the following operations: In response to the portion associated with the reflector, a visual object for verifying the reflector is displayed on the display of the wearable device.

4. The wearable device according to claims 1 to 3, wherein, When executed individually or jointly by the at least one processor, the instructions cause the wearable device to perform the following operations: Identify a first feature point surrounding the portion among the feature points of the image obtained at the first time point; and Based on a second feature point that matches the first feature point and is included in another image obtained at a second time point different from the first time point, another portion of the other image associated with the reflector is identified.

5. The wearable device according to claims 1 to 4, wherein, When executed individually or jointly by the at least one processor, the instructions cause the wearable device to perform the following operations: A second feature point matching the first feature point is identified based on the identifiers assigned to feature points included in the other image.

6. The wearable device according to claims 1 to 5, wherein, When executed individually or jointly by the at least one processor, the instructions cause the wearable device to perform the following operations: By executing instructions for a software application to recreate the virtual space based on the information, at least a portion of the virtual space associated with the external space is displayed on the display of the wearable device.

7. The wearable device according to claims 1 to 6, wherein, When executed individually or jointly by the at least one processor, the instructions cause the wearable device to perform the following operations: Based on the location of the wearable device in external space indicated by the information, the location of the wearable device in virtual space is determined.

8. The wearable device according to claims 1 to 7, wherein, When executed individually or jointly by the at least one processor, the instructions cause the wearable device to perform the following operations: The location of the portion in the image is identified by using a neural network into which the image is input.

9. A method for using a wearable device, comprising: An image of the external space including the wearable device is obtained using the camera of the wearable device (810); The portion of the image associated with the reflector of the reflected light is identified based on object recognition of the image; as well as By using information based on whether each feature point in the image is included in the portion, a software application is performed, providing a virtual space (850) that is at least partially mapped to the external space.

10. The method of claim 9, further comprising: Based on the size of the portion in the image, determine whether at least one feature point identified from the portion is used to identify the position of the wearable device in external space; as well as Based on whether the at least one feature point is used to identify the location, the information indicating the external space is obtained.

11. The method as claimed in claims 9 to 10, wherein, The identification includes: In response to the portion associated with the reflector, a visual object for verifying the reflector is displayed on the display of the wearable device.

12. The method as claimed in claims 9 to 11, wherein, The identification includes: Identify a first feature point surrounding the portion among the feature points of the image obtained at the first time point; and Based on a second feature point that matches the first feature point and is included in another image obtained at a second time point different from the first time point, another portion of the other image associated with the reflector is identified.

13. The method as claimed in claims 9 to 12, wherein, Identifying the other part includes: A second feature point matching the first feature point is identified based on the identifiers assigned to feature points included in the other image.

14. The method as claimed in claims 9 to 13, wherein, The provision includes: By executing instructions for a software application to recreate the virtual space based on the information, at least a portion of the virtual space associated with the external space is displayed on the display of the wearable device.

15. The method as claimed in claims 9 to 14, wherein, The provision includes: Based on the location of the wearable device in external space indicated by the information, the location of the wearable device in virtual space is determined.

Images