Apparatus and method for measuring distance to external object

Abstract

A wearable device may comprise: a memory that stores instructions; at least one processor; a first distance sensor and a second distance sensor; a camera system; and a display system. The instructions, when executed individually or collectively by the at least one processor, may cause the wearable device to: identify the distance to an external object by using images of an external environment; if the distance is greater than or equal to a first reference distance and less than a second reference distance, identify the corrected distance to the external object on the basis of first distance information acquired through the first distance sensor, second distance information acquired through the second distance sensor, and the distance; and display one or more images modified such that a visual object image corresponding to the external object appears to be located at a depth according to the corrected distance.

Description

Device and method for measuring distance to external objects

[0001] The descriptions below relate to devices and methods for measuring distances to external objects.

[0002] To provide an enhanced user experience, an electronic device may provide an extended reality service that displays computer-generated information in conjunction with external objects in the real world or virtual objects in the virtual world. The electronic device may include a wearable device that can be worn by a user. For example, the electronic device may include user equipment, augmented reality (AR) glasses, virtual reality (VR) glasses, and / or a head-mounted device (HMD) (e.g., a video see-through (VST) HMD, an optical see-through (OST) HMD).

[0003] A wearable device may include a memory comprising one or more storage media storing instructions. The wearable device may include at least one processor comprising a processing circuit. The wearable device may include a plurality of sensors, including a first distance sensor and a second distance sensor. The wearable device may include a camera system comprising a plurality of cameras. The wearable device may include a display system, including a first display and a second display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a distance from a reference location to an external object within the external environment using images of the external environment acquired through the camera system. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, when the distance is greater than or equal to a first reference distance and less than a second reference distance that exceeds the first reference distance, first distance information of the external object acquired through the first distance sensor, second distance information of the external object acquired through the second distance sensor, and a corrected distance from the reference location to the external object based on the distance. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display system, one or more images that are modified so that a visual object image corresponding to the external object appears to be located at a depth according to the corrected distance.

[0004] A method performed by a wearable device may include an operation of identifying a distance from a reference position to an external object in the external environment using images of the external environment acquired through a camera system including a plurality of cameras of the wearable device. The method may include an operation of identifying first distance information of the external object acquired through a first distance sensor of the wearable device, second distance information of the external object acquired through a second distance sensor of the wearable device, and a corrected distance from the reference position to the external object based on the distance, when the distance is greater than or equal to a first reference distance and less than a second reference distance that exceeds the first reference distance. The method may include an operation of displaying one or more images that are modified so that a visual object image corresponding to the external object appears to be located at a depth according to the corrected distance.

[0005] A non-transitory computer-readable storage medium may store one or more programs that, when individually or collectively executed by at least one processor of a wearable device including a plurality of sensors including a first distance sensor and a second distance sensor, a camera system including a plurality of cameras, and a display system including a first display and a second display, store instructions that cause the wearable device to identify a distance from a reference location to an external object in the external environment using images of the external environment acquired through the camera system. The non-transitory computer-readable storage medium may store one or more programs that, when individually or collectively executed by the at least one processor, store instructions that cause the wearable device to compare the distance to the external object and measurable distances of the sensors with criteria related to the distances. The non-transitory computer-readable storage medium may store one or more programs that, when individually or collectively executed by the at least one processor, cause the wearable device to identify a corrected distance from the reference position to the external object using the distance and sensor information of the sensors. The non-transitory computer-readable storage medium may store one or more programs that, when individually or collectively executed by the at least one processor, cause the wearable device to display, through the display system, one or more images that are modified such that a visual object image corresponding to the external object appears to be located at a depth according to the corrected distance.

[0006] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0007] FIG. 2A illustrates an example of a perspective view of a wearable device according to various embodiments.

[0008] FIG. 2b illustrates examples of one or more hardware components arranged within a wearable device according to various embodiments.

[0009] FIGS. 3A and 3B illustrate examples of the appearance of a wearable device according to various embodiments.

[0010] Figure 4 illustrates an example of a method for measuring a distance to an external object based on the location of the external object.

[0011] Figure 5 illustrates an exemplary block diagram of a wearable device.

[0012] Figures 6a, 6b, and 6c illustrate examples of a method for measuring a distance to an external object.

[0013] FIG. 7 illustrates an example of an operational flow for a method in which a wearable device identifies a corrected distance based on a distance to an external object measured through an RGB (red-green-blue) camera and displays an image.

[0014] Figure 8a shows an example of a saliency map.

[0015] FIGS. 8B and 8C illustrate examples of a flow of operations for a method for a wearable device to identify a corrected distance using weights based on a saliency map.

[0016] Figure 9 illustrates an example of a speed at which distance is measured by at least one sensor.

[0017] FIG. 10 illustrates an example of an operational flow for a method in which a wearable device uses multiple sensors and at least one camera to identify a calibrated distance to an external object and display an image.

[0018] FIG. 11 illustrates an example of an operational flow for a method of identifying a corrected distance to an external object and displaying an image by comparing a distance to an external object acquired by a wearable device using at least one camera with a plurality of criteria.

[0019] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0020] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0021] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled. However, this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." Conditions described as "more than" may be replaced with "more than," conditions described as "less than," and conditions described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of the elements from A (including A) to B (including B).

[0022] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0023] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0024] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0025] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0026] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0027] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0028] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0029] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0030] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0031] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0032] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0033] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0034] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0036] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0037] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0038] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0040] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0041] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0042] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0043] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0044] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0045] FIG. 2A illustrates an example of a perspective view of a wearable device according to various embodiments. FIG. 2B illustrates an example of one or more hardware components arranged within a wearable device according to various embodiments.

[0046] According to one embodiment, the wearable device (103) may have the form of glasses that are wearable on a body part of the user (e.g., the head). The wearable device (103) of FIGS. 2A and 2B may be an example of the electronic device (101) of FIG. 1. The wearable device (103) may include a head-mounted display (HMD). For example, the housing of the wearable device (103) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (103) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0047] Referring to FIG. 2A, according to one embodiment, a wearable device (103) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0048] According to one embodiment, a wearable device (103) can be worn on a part of a user's body. The wearable device (103) can provide extended reality (XR) to a user wearing the wearable device (103). For example, the extended reality can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality. For example, the wearable device (103) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 263) of FIG. 2B.

[0049] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0050] Referring to FIG. 2B, at least one display (250) can provide the user with visual information transmitted from external light and other visual information distinct from the visual information through a lens included in the at least one display (250). The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When the user wears the wearable device (103), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0051] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (103) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0052] The wearable device (103) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image for various information related to the object included in the real image. The wearable device (103) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (103) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using various time-of-flight (ToF) methods in which a multi-camera (e.g., multiple red-green-blue (RGB) cameras) and / or a depth camera operates. A user wearing the wearable device (103) can view an image displayed on at least one display (250).

[0053] According to one embodiment, the frame (200) may be configured as a physical structure that allows the wearable device (103) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that, when the user wears the wearable device (103), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0054] Referring to FIG. 2A, the frame (200) may include a region (220) that is in contact with at least a portion of a user's body when the user wears the wearable device (103). For example, the region (220) of the frame (200) that is in contact with a portion of the user's body may include a region that is in contact with a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (103) makes contact with. According to one embodiment, the frame (200) may include a nose pad (210) that is in contact with a portion of the user's body. When the wearable device (103) is worn by the user, the nose pad (210) may be in contact with a portion of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another portion of the user's body that is distinct from the portion of the user's body.

[0055] For example, the frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (103) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200).

[0056] According to one embodiment, the wearable device (103) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 5). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290). The various hardwares may be arranged within the frame (200).

[0057] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (103) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B , but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B . When the number of microphones (265) included in the wearable device (103) is two or more, the wearable device (103) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (200).

[0058] According to one embodiment, at least one optical device (282, 284) can project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) can be a projector. At least one optical device (282, 284) can be disposed adjacent to at least one display (250) or can be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (103) can include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0059] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (103). For example, the wearable device (103) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (103) can perform gaze interaction with at least one object using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (103) can express a portion corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (103) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a first region related to the gaze and a second region distinguished from the first region within the image may be different from each other. For example, if the wearable device (103) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1).An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or may be positioned toward both eyes.

[0060] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image in order to implement augmented reality or mixed reality content. The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including an image of the specific object acquired using the capturing camera (260-4) is superimposed on a virtual image provided through at least one optical device (282, 284). The wearable device (103) can compensate for depth information (e.g., the distance between the wearable device (103) and an external object acquired using a depth sensor) using the image acquired using the capturing camera (260-4). The wearable device (103) can perform object recognition using the image acquired using the capturing camera (260-4). The wearable device (103) may perform a pass-through function to display an image acquired through a photographing camera (260-4) by overlaying it on at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the photographing camera (260-4) may be positioned on a bridge (203) positioned between a first rim (201) and a second rim (202).

[0061] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (103) and thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (103) looks straight ahead, the wearable device (103) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (103) is positioned.

[0062] The motion recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The motion recognition camera (260-2, 260-3) can recognize the user's motion (gesture recognition), obtain a signal corresponding to the motion, and provide a display corresponding to the signal on at least one display (250). The wearable device (103) can identify the signal corresponding to the motion, and perform a designated function based on the identification. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The wearable device (103) can perform gesture recognition and / or object tracking functions using motion recognition cameras (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) can be positioned on the first rim (201) and / or the second rim (202).

[0063] The camera (260) included in the wearable device (103) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (103) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (103) can identify an external object based on a sensor for identifying the distance between the wearable device (103) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (103) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (103).

[0064] Although not shown, in one embodiment, the wearable device (103) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (200) and the hinge units (206, 207).

[0065] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (103). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0066] The antenna module (275) can transmit signals or power to the outside of the wearable device (103), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0067] The speaker (255) can output an audio signal to the outside of the wearable device (103). The audio output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (103). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus positioned adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus positioned adjacent to the user's right ear.

[0068] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state, in order to visually provide information regarding a specific state of the wearable device (103) to the user. For example, when the wearable device (103) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0069] Referring to FIG. 2B, according to one embodiment, a wearable device (103) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (103) (e.g., hardwares illustrated by different blocks in FIG. 5) may be disposed on the PCB (290). The wearable device (103) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0070] According to one embodiment, the wearable device (103) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (103) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (103). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (103) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (103) based on the IMU.

[0071] FIGS. 3A and 3B illustrate examples of the appearance of a wearable device according to various embodiments.

[0072] The wearable device (103) of FIGS. 3A and 3B may be an example of the electronic device (101) of FIG. 1. According to one embodiment, an example of the appearance of a first side (310) of a housing of the wearable device (103) may be illustrated in FIG. 3A, and an example of the appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B.

[0073] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (103) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (103) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (103) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0074] According to one embodiment, the wearable device (103) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (103) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (103) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6).

[0075] Referring to FIG. 3b, a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., depth sensors (331, 332)) for obtaining information related to the external environment of the wearable device (103) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3a. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0076] For example, using cameras (260-11, 260-12), the wearable device (103) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be placed on the second face (320) of the wearable device (103) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be placed on the second face (320) of the wearable device (103) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0077] According to one embodiment, the wearable device (103) may include depth sensors (331, 332) disposed on the second face (320) to identify a distance between the wearable device (103) and an external object. For example, the depth sensors (331, 332) may include an indirect-ToF (I-ToF) sensor (331) and a direct-ToF (D-ToF) sensor (332). Using the depth sensors (331, 332), the wearable device (103) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (103). Although not shown, a microphone may be disposed on the second face (320) of the wearable device (103) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0078] The components of the wearable device (103) illustrated in FIGS. 2A to 3B are merely exemplary and the present disclosure is not limited thereto. For example, the wearable device (103) may further include at least one of the components illustrated in FIGS. 2A to 3B or may not include at least one of the components. For example, the wearable device (103) may include the components in a different area (or arrangement) from the area (or arrangement) where the components illustrated in FIGS. 2A to 3B are located. For example, the wearable device (103) may include a different number of components than the number of each of the components (e.g., cameras or sensors) illustrated in FIGS. 2A to 3B.

[0079] The wearable device (103) can detect (or measure, sense, identify, or acquire) a distance to an external object within the external environment using components of the wearable device (103). For example, the external environment may represent an actual environment (or actual environment) representing the surroundings of a user wearing the wearable device (103). For example, the external object may represent an object located within the external environment. For example, the components may include a camera or a sensor.

[0080] For example, the camera may include the capturing camera (260-4) of FIGS. 2A and 2B or the cameras (260-11, 260-12) of FIGS. 3A and 3B. For example, the camera may include at least one RGB camera. The wearable device (103) may detect a distance to the external object using an image (or images) captured (or acquired) using the camera. For example, the distance may represent a straight-line distance from a reference position where a lens of the camera is positioned within the wearable device (103) to the external object. In the example, the reference position is described as corresponding to a position of the lens of the camera within the wearable device (103), but the present disclosure is not limited thereto. For example, the reference position may correspond to a position of a sensor portion (or plane, image plane) within the camera. A specific example of how the wearable device (103) detects the distance using the camera is described below in FIG. 6a.

[0081] For example, the sensor may include the ToF sensor of FIGS. 2A and 2B or the depth sensors (331, 332) of FIGS. 3A and 3B. For example, the sensor may include at least one of an indirect-ToF (I-ToF) sensor or a direct-ToF (D-ToF) sensor. The wearable device (103) may detect the distance using at least one of the I-ToF sensor or the D-ToF sensor. A specific example of a method for the wearable device (103) to detect the distance using the sensor is described below with reference to FIGS. 6B and 6B.

[0082] The wearable device (103) may include at least one of a D-ToF sensor and an I-ToF sensor. For example, the wearable device (103) may detect the distance using distance information acquired using one of the D-ToF sensor and the I-ToF sensor. However, as described below, since the ranges (or areas) in which the D-ToF sensor and the I-ToF sensor can accurately detect the distance are different, the distance to an external object detected by the wearable device (103) may not be accurate. In addition, the D-ToF sensor may take more time to acquire and process distance information than the I-ToF sensor, and the resolution of the sensor may be lower.

[0083] While wearing the wearable device (103), the user may experience a sense of incongruity caused by a difference between the user's field of view (or FoV) and the FoV (or zoom magnification) of the camera of the wearable device (103). For example, the sense of incongruity may be caused by a difference between the depth at which an external object is perceived through the user's eyes and the distance at which it is contacted by a body part (e.g., a hand) of the user. In the above example, the sense of incongruity due to the difference between the user's field of view and the camera's FoV is described, but the present disclosure is not limited thereto. For example, the sense of incongruity may also be caused by a difference between the position of the user's eyes and the position of the camera (or the position of the lens (or the plane of the lens, the image plane)), which is formed by the thickness of the wearable device (103). Alternatively, the above-mentioned discomfort may be caused by a difference between the distance between the user's eyes (interpupillary distance (IPD)) and the distance between the cameras of the wearable device (103).

[0084] To resolve the above-mentioned sense of incongruity, the wearable device (103) may correct an image representing an external environment using the distance information acquired through a D-ToF sensor or an I-ToF sensor. At this time, the depth of a visual object corresponding to the external object in the image may be corrected based on the distance information. For example, the depth of the visual object may represent a positional relationship between other visual objects in the image (e.g., information indicating whether the visual object is located relatively in front or behind). In other words, the wearable device (103) may correct (or map) the image acquired through the camera based on the distance information. For example, the operation of correcting based on the distance information may be referred to as reprojection.

[0085] For example, the reprojection may be performed for each external object, taking into account the distance information acquired by the wearable device (103) and the distance between the wearable device (103) and the user's eyes. For example, the error rate of the distance information for an external object located relatively far from the wearable device (103) (or the user) within the external environment may be different from the error rate of the distance information for an external object located relatively close to the wearable device (103) (or the user). Therefore, the wearable device (103) needs to compensate differently for each external object when identifying the depth of a visual object corresponding to the external object.

[0086] In addition, the resolution of the image acquired through the camera of the wearable device (103) may be higher than the resolution of the distance information (or depth image) for an external object acquired through the sensor of the wearable device (103). Therefore, the wearable device (103) needs to process the distance information acquired through the sensor by up-sampling it at a high magnification to correspond to the high resolution of the image acquired through the camera.

[0087] Hereinafter, the device, method, and storage medium according to the present disclosure may utilize data acquired using a plurality of sensors (e.g., an I-ToF sensor and a D-ToF sensor) and at least one camera (e.g., at least one RGB camera) included in the wearable device (103). For example, the device, method, and storage medium according to the present disclosure may select data to be used for correcting a distance to an external object according to an area where the external object is located within an external environment (or a distance to the external object). The device, method, and storage medium according to the present disclosure may identify a distance to an external object (or a depth of a visual object corresponding to the external object) using at least one of an image acquired through the at least one camera or distance information acquired through the plurality of sensors. The device, method, and storage medium according to the present disclosure may reduce power consumption due to operation of the plurality of sensors by determining at least one sensor among the plurality of sensors to be utilized using the distance to the external object. The device, method, and storage medium according to the present disclosure can perform correction (or reprojection) on an image to be displayed using a more accurately identified (or corrected) distance. Accordingly, the device, method, and storage medium according to the present disclosure can improve the user experience (e.g., reduce the sense of incongruity) by accurately identifying the distance (or depth).

[0088] For convenience of explanation, the present disclosure is described below with reference to a wearable device (103) that provides VST (visual see-through), but the present disclosure is not limited thereto. For example, the present disclosure can also be applied to a wearable device (103) that provides an AR environment.

[0089] Figure 4 illustrates an example of a method for measuring a distance to an external object based on the location of the external object.

[0090] FIG. 4 illustrates an example (400) of a method for a wearable device (103) to measure a distance to external objects (407) based on the location of the external objects (407). The wearable device (103) of FIG. 4 may represent an example of the electronic device (101) of FIG. 1 and the wearable device (103) of FIGS. 2A to 3B.

[0091] Referring to example (400), the wearable device (103) can obtain an image of the external environment (405). For example, the wearable device (103) can obtain the image (or images) of the external environment (405) using at least one camera of the wearable device (103). For example, the image may include an area (410) of the external environment (405) corresponding to the FoV of the at least one camera. For example, the area (410) may be determined based on the FoV and a reference position (420). For example, the reference position (420) may be determined based on a portion where at least one lens of the at least one camera is positioned within the wearable device (103). For example, when the at least one camera includes a plurality of cameras, the reference position (420) may be positioned between a plurality of lenses of the plurality of cameras.

[0092] For example, the wearable device (103) can identify (or obtain) the distance to each of the external objects (407) through the image (or images). For example, the distance can represent the distance from the reference position (420) to each of the external objects (407). For example, the wearable device (103) can identify the distance (a) to the first external object (407-1), the distance (b) to the second external object (407-2), the distance (c) to the third external object (407-3), and the distance (d) to the fourth external object (407-4) by using a distance measurement technique for the image.

[0093] For example, the wearable device (103) may determine a subregion of the external environment (405) in which the identified distance is located. For example, the subregion may be included in a region (410). For example, the region (410) may include a first subregion (411), a second subregion (412), and a third subregion (413). For example, the first subregion (411) may represent a region between a reference location (420) and a first reference distance (421). For example, the first subregion (411) may be referred to as a short-distance region. For example, the second subregion (412) may represent a region between a first reference distance (421) and a second reference distance (422). For example, the second subregion (412) may be referred to as an intermediate distance region. For example, the third partial region (413) may represent a region between the second reference distance (422) and the third reference distance (423). For example, the third partial region (413) may be referred to as a long-distance region. For example, the partial region may be defined by a range between a first reference value (e.g., reference position (420), first reference distance (421), second reference distance (422)) and a second reference value (e.g., first reference distance (421), second reference distance (422), third reference distance (423)) of the distance to an external object. For example, when the first reference value is the reference position (420), the second reference value may be the first reference distance (421). For example, when the first reference value is the first reference distance (421), the second reference value may be the second reference distance (422). For example, if the first reference value is the second reference distance (422), the second reference value may be the third reference distance (423).

[0094] According to one embodiment, the first reference distance (421) may be determined based on a distance measurable by a first distance sensor (e.g., an I-ToF sensor) of the wearable device (103). For example, the I-ToF sensor may be referred to as the first sensor. For example, the measurable distance of the first distance sensor may be from about 10 cm to about 1 m. The examples of the measurable distances are merely examples for convenience of description, and the present disclosure is not limited thereto. For example, the first reference distance (421) may be set to a maximum measurable distance of the first distance sensor (e.g., about 1 m).

[0095] According to one embodiment, the second reference distance (422) may be determined based on a distance measurable by at least one sensor of the wearable device (103) (e.g., at least one camera (e.g., an RGB camera)). For example, the measurable distance of the at least one camera may be from about 50 cm to about 1.5 m. The examples of the measurable distances are merely examples for convenience of description, and the present disclosure is not limited thereto. For example, the second reference distance (422) may be set to a maximum measurable distance of the at least one camera (e.g., about 1.5 m).

[0096] According to one embodiment, the third reference distance (423) may be determined based on a distance measurable by a second distance sensor (e.g., a D-ToF sensor) of the wearable device (103). For example, the D-ToF sensor may be referred to as the second sensor. For example, the measurable distance of the second distance sensor may be from about 20 cm to about 5 m. The examples of the measurable distances are merely examples for convenience of description, and the present disclosure is not limited thereto. For example, the third reference distance (423) may be set to a maximum measurable distance of the second distance sensor (e.g., about 5 m).

[0097] For example, the measurable distance may represent a measurement range having an accuracy higher than a reference accuracy of the at least one camera, the first distance sensor, and the second distance sensor. In other words, the wearable device (103) is not incapable of measuring a distance to an external object located at a distance outside the measurable distance using the at least one camera, the first distance sensor, and the second distance sensor. The accuracy (or reliability) of the measured distance to an external object located at a distance outside the measurable distance may be lower than the reference accuracy. For example, the measurable distance having an accuracy higher than the reference accuracy may include a distance in which an error between an estimated distance to an external object and an actual distance to the external object is less than a reference difference.

[0098] In the above example, an example using the first distance sensor (e.g., an I-ToF sensor), the at least one camera, and the second distance sensor (e.g., a D-ToF sensor) is described, but the present disclosure is not limited thereto. For example, the at least one camera may be replaced with an image sensor. In addition, the measurable distance may vary depending on the sensor used. For example, the measurable distance of the first distance sensor may differ from the measurable distance of the second distance sensor. In addition, the accuracy of determining the measurable distance may vary depending on the sensor. In the present disclosure, an example using three components (or sensors) (e.g., the first distance sensor, the at least one camera, and the second distance sensor) is described, but the present disclosure is not limited thereto. For example, the present disclosure can also be applied to a wearable device (103) including four components (or sensors). According to one embodiment, the wearable device (103) may utilize three or more components to perform operations according to the present disclosure, and may utilize a variable number of components while performing the operations.

[0099] Referring to the above, the first partial region (411) may represent a region of the external environment (405) between the reference position (420) (e.g., 0) and the first reference distance (421) (e.g., about 1 m). The second partial region (412) may represent a region of the external environment (405) between the first reference distance (421) (e.g., about 1 m) and the second reference distance (422) (e.g., about 1.5 m). The third partial region (413) may represent a region of the external environment (405) between the second reference distance (422) (e.g., about 1.5 m) and the third reference distance (423) (e.g., about 5 m).

[0100] In the above example, the region (410) is illustrated as including a first sub-region (411), a second sub-region (412), and a third sub-region (413), but the embodiments of the present disclosure are not limited thereto. For example, the region (410) may include a region (or remaining region) located at a distance greater than the third sub-region (413) (e.g., a region representing a distance of about 5 m or more). Or, for example, the region (410) may include a region (or region of interest) where the gaze (or direction of the gaze) of the user of the wearable device (103) is located. For example, the region of interest may represent a region defined across at least one sub-region. Or, for example, the region (410) may further include a fourth sub-region (414) included in the first sub-region (411). For example, the fourth sub-region (414) may represent a region (e.g., about 50 cm to about 1 m) extending from a portion (424) of the first sub-region (411) toward the second sub-region (412) to a first reference distance (421). For example, the fourth sub-region (414) may be determined based on a minimum measurable distance (e.g., about 50 cm) of the at least one camera and a maximum measurable distance (e.g., about 1 m) of the first distance sensor (or the first reference distance (421)).

[0101] For example, the wearable device (103) can determine that the first external object (407-1) is located within the first partial region (411) based on the distance (a) identified through the image. The wearable device (103) can determine that the second external object (407-2) is located within the second partial region (412) based on the distance (b) identified through the image. The wearable device (103) can determine that the third external object (407-3) is located within the third partial region (413) based on the distance (c) identified through the image. The wearable device (103) can determine that the fourth external object (407-4) is located within the remaining region of the region (410) based on the distance (d) identified through the image.

[0102] For example, the wearable device (103) may identify a corrected distance from the distance to the external object using at least one of the first distance sensor and the second distance sensor, depending on the distance to the external object (or the area in which the external object is located within the external environment (405)). For example, the corrected distance may be referred to as the final distance. For example, when the distance (e.g., distance (a)) to the external object (e.g., the first external object (407-1)) is less than the first reference distance (421), the wearable device (103) may identify the corrected distance (a`) from the reference position (420) to the external object based on the first distance information acquired from the first distance sensor and the distance to the external object (or the distance identified through an image acquired from the at least one camera). For example, the wearable device (103) can identify a corrected distance (b`) from a reference position (420) to the external object based on the first distance information acquired from the first distance sensor, the second distance information acquired from the second distance sensor, and the distance to the external object when the distance (e.g., distance (b)) to an external object (e.g., a second external object (407-2)) is greater than or equal to a first reference distance (421) and less than a second reference distance (422). For example, the wearable device (103) can identify a corrected distance (c`) from a reference position (420) to the external object based on the second distance information acquired from the second distance sensor and the distance to the external object when the distance (e.g., distance (c)) to an external object (e.g., a third external object (407-3)) is greater than or equal to a second reference distance (422) and less than a third reference distance (423).

[0103] In one embodiment, the wearable device (103) may correct (or reproject) the image acquired through the at least one camera based on the corrected distance. For example, the wearable device (103) may generate a corrected image so that a visual object corresponding to an external object appears (or is recognized) to be located at a depth according to the corrected distance. For example, the wearable device (103) may display the corrected image through at least one display of the wearable device (103).

[0104] According to one embodiment, the wearable device (103) may generate an image so that a visual object corresponding to the external object appears (or is recognized) to be located at a distance (e.g., distance (d)) from an external object (e.g., a fourth external object (407-4)) greater than or equal to a third reference distance (423). For example, the wearable device (103) may display the generated image through at least one display of the wearable device (103). In other words, the wearable device (103) may refrain from (or skip, delay, or exclude) performing correction of the image for external objects located in the remaining area.

[0105] Although not illustrated in the example (400) of FIG. 4, according to one embodiment, the wearable device (103) may identify the corrected distance further based on a saliency map. For example, the wearable device (103) may calculate a weight to be applied to the distance acquired using the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor, based on the saliency map including information on the user's area of ​​interest. For example, the weight may be referred to as a weight value, a compensation value, or a compensation ratio. For example, the area of ​​interest may be referred to as a weight area. For example, the wearable device (103) may adjust the operating ratio between the at least one camera and sensors by applying the calculated weight to the distance, the first distance information, and the second distance information. Additionally, for example, the wearable device (103) can identify the region of interest by further utilizing the area (or gaze range) toward which the user's gaze is directed, along with the protrusion map. Specific details related to this are described below in FIGS. 8A to 8C.

[0106] According to one embodiment, the wearable device (103) may identify the corrected distance based on scene analysis of the image acquired through the at least one camera. For example, the scene analysis may include analysis based on SLAM. For example, the wearable device (103) may identify the location of each of one or more external objects within the image based on the scene analysis. For example, the wearable device (103) may store information about the location of each of the one or more external objects as scene understanding data. For example, the scene understanding data may be utilized in a software application executed by the wearable device (103). For example, the scene understanding data may be utilized for a service provided by the software application. When the scene understanding data is stored in a number greater than or equal to a reference data number, the wearable device (103) may recognize that one or more objects of interest are located within a specific sub-region among the sub-regions. For example, the wearable device (103) may store information about a location identified from each image as the scene understanding data. If the number of the scene understanding data stored is greater than or equal to the number of reference data, the wearable device (103) may recognize that one or more objects of interest are located within a specific sub-region among the sub-regions. For example, the wearable device (103) may identify the specific sub-region in which the one or more objects of interest are located as the region of interest (or weighted region). For example, the object of interest may include an external object of interest to the user.Accordingly, the wearable device (103) can calculate weights to be applied to the distance acquired through the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor. For example, the wearable device (103) can adjust the operating ratio between the sensors by applying the calculated weights to the distance, the first distance information, and the second distance information.

[0107] According to one embodiment, the wearable device (103) may identify the corrected distance based on a user's gesture. For example, the wearable device (103) may identify a gesture or interaction by the user's hand through a sensor or a camera. When the wearable device (103) identifies the gesture or interaction, the wearable device (103) may identify the first sub-region (411) as a region of interest, and accordingly calculate a weight to be applied to the distance acquired through the image and the first distance information of the first distance sensor. However, the present disclosure is not limited thereto. For example, when the wearable device (103) identifies the gesture or interaction, the wearable device (103) may identify the fourth sub-region (414) within the first sub-region (411) as a region of interest, and calculate a weight to be applied to the distance acquired through the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor. Alternatively, for example, when the wearable device (103) identifies the gesture or interaction, it can identify the first partial area (411) as a region of interest and calculate weights to be applied to the distance acquired through the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor. In this case, for higher accuracy even within the first partial area (411), the wearable device (103) can identify a corrected distance to an external object using the second distance information together with the first distance information.

[0108] According to one embodiment, when the illuminance of the external environment is lower than the reference illuminance, the wearable device (103) may identify the distance to an external object in the external environment based on the second distance information of the second distance sensor instead of the image acquired from the at least one camera. By comparing the distance to the external object identified based on the second distance information with the partial regions, the wearable device (103) may determine the distance information to be used.

[0109] According to one embodiment, when the surface of the external object is made of a material that reflects light, the wearable device (103) may calculate a weight to be applied to an image acquired through the at least one camera (or a distance acquired using the image) as a relatively higher value compared to the distance information acquired from the first distance sensor and the second distance sensor. In addition, when processing an area in the external environment where the light is scattered (e.g., a periphery (or corner) of the external object), the wearable device (103) may calculate a weight to be applied to an image acquired through the at least one camera (or a distance acquired using the image) as a relatively higher value compared to the distance information acquired from the first distance sensor and the second distance sensor. Accordingly, the wearable device (103) may prevent malfunction of the ToF sensors and / or correct specific conditions.

[0110] Figure 5 illustrates an exemplary block diagram of a wearable device.

[0111] The wearable device (103) of FIG. 5 may be an example of the electronic device (101) of FIG. 1, the wearable device (103) of FIGS. 2A to 3B, or the wearable device (103) of FIG. 4.

[0112] Referring to FIG. 5, an exemplary situation is illustrated in which a wearable device (103) is connected to an external electronic device (not shown) based on a wired network and / or a wireless network. For example, the wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), or a combination thereof. For example, the wireless network may include a network such as long term evolution (LTE), 5g new radio (NR), wireless fidelity (Wi-Fi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. Although the wearable device (103) and the external electronic device are illustrated as being directly connected, the wearable device (103) and the external electronic device may be indirectly connected via one or more routers and / or access points (APs).

[0113] Referring to FIG. 5, according to one embodiment, a wearable device (103) may include at least one of a processor (510), a display (520), a sensor (530), a camera (540), and a memory (550). The processor (510), the display (520), the sensor (530), the camera (540), and the memory (550) may be electrically and / or operably coupled with each other by a communication bus. Hereinafter, operably coupled hardware components may mean that a direct connection or an indirect connection is established between the hardware components, either wired or wireless, such that a second hardware component is controlled by a first hardware component among the hardware components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware components illustrated in FIG. 5 (e.g., at least a portion of the processor (510) and the memory (550)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware components included in the wearable device (103) is not limited to those illustrated in FIG. 5. For example, the wearable device (103) may include only some of the hardware components illustrated in FIG. 5.

[0114] According to one embodiment, the processor (510) of the wearable device (103) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), and / or a neural processing unit (NPU). The number of processors (510) may be one or more. For example, the processor (510) may have a multi-core processor structure, such as a dual core, a quad core, or a hexa core. The processor (510) of FIG. 5 may include the processor (120) of FIG. 1.

[0115] For example, the processor (510) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform the various functions.

[0116] According to one embodiment, a display (520) of a wearable device (103) may output visualized information to a user. The number of displays (520) included in the wearable device (103) may be one or more. For example, the display (520) may be referred to as a display system including one or more displays. For example, the display (520) may be controlled by a processor (510) and / or a graphic processing unit (GPU) (not shown) to output visualized information to a user. The display (520) may include a flat panel display (FPD) and / or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), a digital mirror device (DMD), one or more light emitting diodes (LEDs), and / or micro LEDs. The LEDs may include organic LEDs (OLEDs). The display (520) of FIG. 5 may include the display module (160) of FIG. 1.

[0117] According to one embodiment, the wearable device (103) may include a sensor (530). For example, the sensor (530) may include ToF sensors for measuring a distance to an external object located in an external environment around the wearable device (103). For example, the ToF sensors may include an indirect-ToF (I-ToF) sensor (or a first distance sensor) and a direct-ToF (D-ToF) sensor (or a second distance sensor). For example, the wearable device (103) may generate (or obtain) distance information indicating the distance using at least one ToF sensor among the ToF sensors. For example, distance information obtained through the I-ToF sensor may be referred to as first distance information. For example, distance information obtained through the D-ToF sensor may be referred to as second distance information. Specific examples of how the wearable device (103) detects the distance using the sensor (530) are described below in FIGS. 6B and 6C . However, the present disclosure is not limited thereto. For example, the wearable device (103) may include at least one sensor for detecting a user's gesture. For example, the sensor (530) of FIG. 5 may include at least a portion of the sensor module (176) of FIG. 1 .

[0118] In one embodiment, the wearable device (103) may include a camera (540). The camera (540) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. For example, the camera (540) may be used as (or replaced with) an image sensor. For example, the camera (540) may be referred to as a camera system including one or more cameras. The plurality of optical sensors included in the camera may be arranged in the form of a two-dimensional array. The camera (540) may acquire electrical signals from each of the plurality of optical sensors substantially simultaneously, and generate an image corresponding to light reaching the optical sensors of the two-dimensional array, the image including a plurality of pixels arranged two-dimensionally. According to one embodiment, the wearable device (103) can render an actual environment within the image obtained through the camera (540) and display the rendered visual information. The number of cameras (540) included in the wearable device (103) may be one or more, as described above with reference to FIGS. 2A and 2B and / or FIGS. 3A and 3B. For example, the camera (540) may be referred to as an RGB camera or an RGB vision camera.

[0119] According to one embodiment, the memory (550) of the wearable device (103) may include a hardware component for storing data and / or instructions input to and / or output from the processor (510). The memory (550) may include, for example, a volatile memory such as a random-access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multimedia card (eMMC). The memory (550) of FIG. 5 may include the memory (130) of FIG. 1.

[0120] Although not illustrated in FIG. 5, the wearable device (103) may include a communication circuit. For example, the communication circuit may include hardware for supporting transmission and / or reception of electrical signals between the wearable device (103) and an external electronic device. The communication circuit may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). The communication circuit may include the communication module (190) and / or the antenna module (197) of FIG. 1.

[0121] Although not illustrated in FIG. 5, the wearable device (103) according to one embodiment may include an output means for outputting information in a form other than a visualized form. For example, the wearable device (103) may include a speaker for outputting an acoustic signal. For example, the wearable device (103) may include a motor for providing haptic feedback based on vibration.

[0122] Referring to FIG. 5, according to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (510) of the wearable device (103) may be stored in the memory (550) of the wearable device (103). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, when an application is installed in an electronic device (e.g., a wearable device (103)), it may mean that one or more instructions provided in the form of an application are stored in the memory (550), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension designated by the operating system of the wearable device (103)). According to one embodiment, the wearable device (103) may perform the operations of FIGS. 7, 8b, 8c, and 9 by executing one or more instructions stored in the memory (550).

[0123] For example, programs installed in the wearable device (103) may be classified into one of different layers, including an application layer, a framework layer, and / or a hardware abstraction layer (HAL), based on the target. For example, programs (e.g., drivers) designed to target the hardware of the wearable device (103) (e.g., the display (520), the sensor (530), and / or the camera (540)) may be classified within the hardware abstraction layer. For example, programs (e.g., a distance information generation module (551), a reference distance setting module (553), a distance correction module (555), and / or an image correction module (557)) designed to target at least one of the hardware abstraction layer and / or the application layer may be classified within the framework layer. Programs classified into the framework layer may provide an executable API (Application Programming Interface) based on other programs.

[0124] For example, within the application layer, programs designed to target users controlling wearable devices (103) may be classified. For example, a program classified as the application layer may include at least one application that provides an XR environment. However, embodiments of the present disclosure are not limited thereto. For example, a program classified as the application layer may call an API to cause the execution of functions supported by programs classified as the framework layer.

[0125] Referring to FIG. 5, the wearable device (103) can detect (or identify, acquire, measure) a distance to an external object in the external environment based on the execution of the distance information generation module (551). For example, the wearable device (103) can detect the distance to the external object using an image (or images) captured (or acquired) using the camera (540). The distance detected (or identified) using the image may be referred to as a temporary distance or an initial distance. For example, the wearable device (103) can acquire the first distance information (or depth image) indicating the distance to the external object using the first distance sensor. Alternatively, for example, the wearable device (103) can acquire the second distance information (or depth image) indicating the distance to the external object using the second distance sensor. For example, the depth image may include an image for visually displaying the depth of the external object.

[0126] Referring to FIG. 5, the wearable device (103) can set a reference distance based on the execution of the reference distance setting module (553). For example, the wearable device (103) can set a plurality of reference distances based on the capabilities (or performances) of the sensor (530) and / or the camera (540). For example, the plurality of reference distances can represent criteria for distinguishing a plurality of partial regions (e.g., the first partial region (411), the second partial region (412), and the third partial region (413) of FIG. 4) included in a region of the external environment around the wearable device (103) (e.g., the region (410) of FIG. 4). For example, the plurality of reference distances may include a first reference distance determined based on a measurable distance of the first distance sensor (e.g., the first reference distance (421) of FIG. 4), a second reference distance determined based on a measurable distance of the camera (540) (e.g., the second reference distance (422) of FIG. 4), and a third reference distance determined based on a measurable distance of the second distance sensor (e.g., the third reference distance (423) of FIG. 4). However, the present disclosure is not limited thereto. In the example, the wearable device (103) may also change (or adjust) each of the plurality of reference distances based on at least a portion of a user's input.

[0127] Referring to FIG. 5, the wearable device (103) may correct a distance to an external object based on the execution of the distance correction module (555). For example, the wearable device (103) may identify a corrected distance based on the distance identified from the image acquired through the camera (540) and at least one piece of distance information. For example, the at least one piece of distance information may be determined based on a comparison between the distance identified from the image and the plurality of reference distances. For example, when the distance is less than the first reference distance, the at least one piece of distance information may include the first distance information. For example, when the distance is less than the second reference distance and greater than or equal to the first reference distance, the at least one piece of distance information may include the first distance information and the second distance information. For example, when the distance is greater than or equal to the second reference distance, the at least one piece of distance information may include the second distance information. However, the present disclosure is not limited thereto.

[0128] According to one embodiment, the wearable device (103) can identify (or generate) the corrected distance that is changed (or adjusted) from the distance using the at least one piece of distance information. For example, the wearable device (103) can identify the corrected distance (x`) from the distance (x) using the at least one piece of distance information. For example, the corrected distance (x`) can be identified based on a formula (x`=(x+α) / n) in which a correction value (α) calculated from the at least one piece of distance information is added to the distance (x). The value (n) can represent the number of the distance and the at least one piece of distance information used. For example, the correction value (α) can be calculated based on a formula (e.g., α=a1I+a2D). For example, the value (I) may represent a first value based on the first distance information, the value (D) may represent a second value based on the second distance information, the value (a1) may represent a first weight to be applied to the first distance information, and the value (a2) may represent a second weight to be applied to the second distance information. However, the above formulas are merely exemplary, and the present disclosure is not limited thereto. For example, the corrected distance (x`) may be identified based on the formula (x`=(a3x+α) / n). For example, the value (a3) ​​may represent a third weight to be applied to the distance. The calculation method described above may be referred to as a normalized method.

[0129] Alternatively, the wearable device (103) may identify (or generate) the corrected distance based on the at least one piece of distance information determined to be used according to the distance. For example, the wearable device (103) may determine the at least one piece of distance information to be used based on the distance, and identify the corrected distance using the determined at least one piece of distance information. For example, when the distance is greater than or equal to the first reference distance and less than the second reference distance, the wearable device (103) may determine to use the first distance information and the second distance information to identify the corrected distance. For example, the wearable device (103) may calculate the corrected distance (x`) using the first value (I) based on the first distance information and the second value (D) based on the second distance information. For example, the corrected distance (x`) can be calculated through interpolation by the first value (I) and the second value (D). The calculation method as described above can be referred to as an interpolation method. For example, according to the interpolation method, when calculating the corrected distance (x`), a first weight to be applied to the first value and a second weight to be applied to the second value can be used. The first weight and the second weight are used as factors that are substantially the same as the first weight and the second weight of the normalization method, but can have different values ​​from the first weight and the second weight of the normalization method.

[0130] Referring to FIG. 5, the wearable device (103) may correct an image to be displayed based on the execution of the image correction module (557). For example, the wearable device (103) may generate a corrected image from the image acquired through the camera (540). For example, the corrected image may be changed (or adjusted) so that a visual object corresponding to an external object appears (or is recognized) to be located at a depth according to the corrected distance. As described above, the wearable device (103) may perform correction (or reprojection) for each of one or more visual objects within the image. For example, the one or more visual objects may represent visual information corresponding to one or more external objects within the external environment.

[0131] Figures 6a to 6c illustrate examples of a method for measuring a distance to an external object.

[0132] FIG. 6A illustrates an example (600) of a method for measuring a distance to an external object (605) using cameras (601, 602) of a wearable device (103). The cameras (601, 602) of FIG. 6A may be an example of the camera (540) of FIG. 5 . Although the example (600) of FIG. 6A illustrates an example of a method for measuring the distance using two cameras (601, 602), the present disclosure is not limited thereto. For example, the wearable device (103) may also measure the distance using a single camera.

[0133] Referring to example (600), the wearable device (103) can acquire images using cameras (601, 602) positioned at different locations, respectively, and identify the distance to the external object (605) based on the degree of parallax (or binocular parallax) of the visual object corresponding to the external object (605) in each image. For example, the shorter the distance to the external object (605) (or the closer the distance from the wearable device (103) (or the reference location) to the external object (605)), the greater the parallax of the visual object between the images. For example, the parallax can be determined based on a comparison between feature points for the visual object in each of the images. For example, assume that the images include a first image acquired through the camera (601) and a second image acquired through the camera (602). A difference between first feature points for a visual object in the first image and second feature points for a visual object in the second image can be identified. For example, the difference can be defined by the number of pixels. For example, a smaller number of pixels can indicate that the visual object is located at a relatively farther distance, and a larger number of pixels can indicate that the visual object is located at a relatively closer distance.

[0134] In one embodiment, the distance may be identified further based on at least one of a separation distance (607) between cameras (601, 602), a separation distance (609) between the eyes of a user of the wearable device (103), or a separation distance (611) between a camera and an eye. For example, the distance may be identified further based on at least one of the separation distance (607), the separation distance (609), or the separation distance (611) along with the difference.

[0135] For example, the wearable device (103) can measure a distance to an external object (605) within about 50 cm to about 1.5 m using the cameras (601, 602). In this case, the range of about 50 cm to about 1.5 m may represent a measurable distance. The measurable distance may represent a measurement range having an accuracy higher than the standard accuracy of the cameras (601, 602). In other words, the wearable device (103) is not incapable of measuring a distance to an external object located at a distance outside the measurable distance using the cameras (601, 602). The accuracy (or reliability) of the measured distance to an external object located at a distance outside the measurable distance may be lower than the standard accuracy. For example, the image acquired by the wearable device (103) using the cameras (601, 602) may have a relatively high resolution (e.g., 3000x3000) compared to the sensor information (or depth image) acquired using the first distance sensor (621) of FIG. 6B and the sensor information (or depth image) acquired using the second distance sensor (641) of FIG. 6C. For example, the wearable device (103) may acquire (or capture) the image at a relatively fast speed (e.g., 90 fps (frame per second)) using the cameras (601, 602) compared to the sensor information (or depth image) acquired using the first distance sensor (621) of FIG. 6B and the sensor information (or depth image) acquired using the second distance sensor (641) of FIG. 6C.

[0136] FIG. 6B illustrates an example (620) of a method for measuring a distance to an external object (625) using a first distance sensor (621) of a wearable device (103). The first distance sensor (621) of FIG. 6B may be included in the sensor (530) of FIG. 5 . For example, the first distance sensor (621) may be an example of an I-ToF sensor.

[0137] Referring to example (620), the wearable device (103) can emit light (627-1) toward an external object (625) via a first distance sensor (621). For example, the light (627-1) can be emitted via an emitter (622) of the first distance sensor (621). For example, the light (627-1) can include modulated light. For example, the wearable device (103) can receive light (627-2) reflected by the external object (625). For example, the light (627-2) can be received via a receiver (623). For example, the light (627-2) can have characteristics that are changed from the light (627-1). For example, the phase of light (627-2) may be different from the phase of light (627-1). The wearable device (103) can identify the distance to an external object (625) based on the phase difference between lights (627-1, 627-2).

[0138] For example, the wearable device (103) can measure a distance to an external object (625) within about 10 cm to about 1 m using the first distance sensor (621). In this case, the range of about 10 cm to about 1 m may represent a measurable distance. The measurable distance may represent a measurement range having an accuracy higher than the standard accuracy of the first distance sensor (621). In other words, the wearable device (103) is not incapable of measuring a distance to an external object located at a distance outside the measurable distance using the first distance sensor (621). The accuracy (or reliability) of the measured distance to an external object located at a distance outside the measurable distance may be less than the standard accuracy. For example, the first distance information acquired by the wearable device (103) using the first distance sensor (621) may include a depth image having a relatively medium resolution (e.g., 320x240) compared to the images acquired using the cameras (601, 602) of FIG. 6A and the sensor information (or depth image) acquired using the second distance sensor (641) of FIG. 6C. For example, the wearable device (103) may acquire (or capture) the depth image using the first distance sensor (621) at a relatively medium speed (e.g., 30 fps (frames per second)) compared to the images acquired using the cameras (601, 602) of FIG. 6A and the sensor information (or depth image) acquired using the second distance sensor (641) of FIG. 6C.

[0139] FIG. 6C illustrates an example (640) of a method for measuring a distance to an external object (645) using a second distance sensor (641) of a wearable device (103). The second distance sensor (641) of FIG. 6C may be included in the sensor (530) of FIG. 5. For example, the second distance sensor (641) may be an example of a D-ToF sensor.

[0140] Referring to example (640), the wearable device (103) can emit light (647) toward an external object (645) through a second distance sensor (641). For example, the light (647) can be emitted through an emitter (642) of the second distance sensor (641). For example, the light (647) can include an infrared laser (IR laser) in a light pulse format. For example, the wearable device (103) can receive the light (647) by an external object (645). For example, the reflected light (647) can be received through a receiver (643). For example, the wearable device (103) can identify the distance to an external object (645) by measuring the time between the timing at which light (647) is emitted and the timing at which light (647) is received.

[0141] For example, the wearable device (103) can measure a distance to an external object (645) within about 20 cm to about 5 m using the second distance sensor (641). In this case, the range of about 20 cm to about 5 m may represent a measurable distance. The measurable distance may represent a measurement range having an accuracy higher than the standard accuracy of the second distance sensor (641). In other words, the wearable device (103) is not incapable of measuring a distance to an external object located at a distance outside the measurable distance using the second distance sensor (641). The accuracy (or reliability) of the measured distance to an external object located at a distance outside the measurable distance may be less than the standard accuracy. For example, the second distance information acquired by the wearable device (103) using the second distance sensor (641) may include a depth image having a relatively low resolution (e.g., 24x24) compared to the images acquired using the cameras (601, 602) of FIG. 6A and the sensor information (or depth image) acquired using the first distance sensor (621) of FIG. 6B. For example, the wearable device (103) may acquire (or capture) a depth image using the second distance sensor (641) at a relatively low speed (e.g., 10 fps (frames per second)) compared to the images acquired using the cameras (601, 602) of FIG. 6A and the sensor information (or depth image) acquired using the first distance sensor (621) of FIG. 6B.

[0142] Referring to FIGS. 6A to 6C , the wearable device (103) can identify the distance to an external object by using a relatively large number (or relatively frequently acquired) of images with a relatively high resolution compared to the first distance sensor (621) and the second distance sensor (641) acquired using the cameras (601, 602). At this time, for more accurate distance measurement, the wearable device (103) can further use the first distance sensor (621) and the second distance sensor (641). For example, the first distance sensor (621) can acquire a large number of distance information (or depth images) with a relatively high resolution compared to the second distance sensor (641). However, the first distance sensor (621) can acquire distance information for an external object at a relatively close distance compared to the second distance sensor (641).

[0143] In addition, if the external environment has an illuminance value lower than the reference illuminance, the distance identified using the images acquired through the cameras (601, 602) may have low accuracy (or reliability). Accordingly, the wearable device (103) may use the first distance sensor (621) or the second distance sensor (641). Conversely, if the surface of the external object is made of a material that reflects or absorbs light, or if diffuse reflection is caused (e.g., the corner / edge of the external object), the distance identified using the distance information (or depth image) acquired through the sensors (621, 641) may have low accuracy (or reliability). Accordingly, the wearable device (103) may use the cameras (601, 602).

[0144] FIG. 7 illustrates an example of an operational flow for a method in which a wearable device identifies a corrected distance based on a distance to an external object measured through an RGB (red-green-blue) camera and displays an image.

[0145] At least some of the methods of FIG. 7 may be performed by the wearable device (103) of FIG. 5. For example, at least some of the methods may be controlled by the processor (510) of the wearable device (103). In the embodiments described below, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0146] In operation (700), the wearable device (103) may acquire an image through a camera (e.g., an RGB camera). For example, the RGB camera may be included in the camera (540) of FIG. 5. For example, the RGB camera may include at least one RGB camera. For example, the wearable device (103) may acquire an image of an area including an external object in an external environment. For example, the image may represent visual information representing the area of ​​the external environment. For example, the image may include a visual object corresponding to the external object. For example, a period of acquiring an image through the RGB camera may have a first time interval (e.g., a frame interval of 90 fps).

[0147] In operation (705), the wearable device (103) may obtain a distance to the external object using the acquired image. For example, the distance may be referred to as a temporary distance or an initial distance. For example, when the wearable device (103) uses multiple RGB cameras, the distance may be obtained based on a parallax between images acquired from the multiple RGB cameras. For example, the parallax between the images may represent the degree of parallax (or binocular parallax) of a visual object corresponding to the external object in each image. For example, the parallax may be determined based on a comparison between feature points for the visual object in each of the images. However, the present disclosure is not limited thereto. For example, the parallax may also be determined based on a separation distance between the multiple RGB cameras.

[0148] According to one embodiment, the wearable device (103) may determine a sub-region of the external environment where the distance is located. For example, the sub-region may be included in the area. For example, the area may include a plurality of sub-regions. For example, the plurality of sub-regions may include a first sub-region, a second sub-region, and a third sub-region. For example, the first sub-region may represent an area between a reference location and a first reference distance. For example, the first sub-region may be referred to as a near-distance area. For example, the second sub-region may represent an area between a first reference distance and a second reference distance. For example, the second sub-region may be referred to as an intermediate-distance area. For example, the third sub-region may represent an area between a second reference distance and a third reference distance. For example, the third sub-region may be referred to as a far-distance area. For example, the sub-region may be defined by a range of distances to an external object between a first reference value (e.g., reference position, first reference distance, second reference distance) and a second reference value (e.g., first reference distance, second reference distance, third reference distance).

[0149] According to one embodiment, the first reference distance may be determined based on a distance measurable by a first distance sensor of the wearable device (103) (e.g., the first distance sensor (621) of FIG. 6B ). For example, the measurable distance of the first distance sensor may be from about 10 cm to about 1 m. The above example is merely an example for convenience of explanation, and the present disclosure is not limited thereto. For example, the first reference distance may be set to a maximum measurable distance of the first distance sensor (e.g., about 1 m).

[0150] According to one embodiment, the second reference distance may be determined based on a distance measurable by a camera (e.g., an RGB camera) of the wearable device (103). For example, the measurable distance of the RGB camera may be from about 50 cm to about 1.5 m. The above example is merely an example for convenience of explanation, and the present disclosure is not limited thereto. For example, the second reference distance may be set to a maximum measurable distance of the at least one camera (e.g., about 1.5 m). For example, the second reference distance may have a value greater than the first reference distance.

[0151] According to one embodiment, the third reference distance may be determined based on a distance measurable by the second distance sensor of the wearable device (103). For example, the measurable distance of the second distance sensor may be from about 20 cm to about 5 m. The above example is merely an example for convenience of explanation, and the present disclosure is not limited thereto. For example, the third reference distance may be set to a maximum measurable distance of the second distance sensor (e.g., about 5 m). For example, the third reference distance may have a value greater than the second reference distance.

[0152] According to one embodiment, the measurable distance may represent a measurement range having an accuracy higher than the reference accuracy of the RGB camera, the first distance sensor, and the second distance sensor. In other words, the wearable device (103) is not incapable of measuring a distance to an external object located at a distance outside the measurable distance using at least one camera, the first distance sensor, and the second distance sensor. The accuracy (or reliability) of the measured distance to an external object located at a distance outside the measurable distance may be lower than the reference accuracy.

[0153] In operation (710), the wearable device (103) may determine whether the distance is less than a second reference distance. In operation (710), if the distance to the external object is less than the second reference distance, the wearable device (103) may perform operation (715). Conversely, in operation (710), if the distance to the external object is greater than or equal to the second reference distance, the wearable device (103) may perform operation (730).

[0154] In operation (715), the wearable device (103) may determine whether the distance is less than a first reference distance. For example, if the distance to the external object is less than a second reference distance, the wearable device (103) may determine whether the distance to the external object is less than the first reference distance. If the distance to the external object is less than the first reference distance in operation (715), the wearable device (103) may perform operation (720). Alternatively, if the distance to the external object is greater than or equal to the second reference distance in operation (715), the wearable device (103) may perform operation (725).

[0155] In operation (720), the wearable device (103) may identify a corrected distance based on the first distance information acquired through the first distance sensor and the distance to the external object. For example, if the distance to the external object is less than the first reference distance, the wearable device (103) may identify a corrected distance based on the first distance information and the distance to the external object. For example, the first distance information may include a depth image acquired through the first distance sensor. For example, the period in which the first distance information acquired through the first distance sensor is acquired may have a second time interval (e.g., a frame interval of 30 fps) that is longer than the first time interval.

[0156] In operation (725), the wearable device (103) can identify a corrected distance based on first distance information acquired through the first distance sensor, second distance information acquired through the second distance sensor, and the distance to the external object. For example, when the distance to the external object is greater than or equal to the first reference distance and less than the second reference distance, the wearable device (103) can identify a corrected distance based on the first distance information, the second distance information, and the distance to the external object. For example, the second distance information may include a depth image acquired through the second distance sensor. For example, the period in which the second distance information acquired through the second distance sensor is acquired may have a third time interval (e.g., a frame interval of 10 fps) that is longer than the second time interval.

[0157] In operation (730), the wearable device (103) may determine whether the distance to the external object is less than a third reference distance. For example, if the distance to the external object is greater than or equal to the second reference distance, the wearable device (103) may determine whether the distance to the external object is less than or equal to the third reference distance. In operation (730), if the distance to the external object is less than the third reference distance, the wearable device (103) may perform operation (735). Alternatively, in operation (730), if the distance to the external object is greater than or equal to the third reference distance, the wearable device (103) may perform operation (745).

[0158] In operation (735), the wearable device (103) can identify a corrected distance based on the second distance information acquired through the second distance sensor and the distance to the external object. For example, if the distance to the external object is greater than or equal to the second reference distance and less than the third reference distance, the wearable device (103) can identify a corrected distance based on the second distance information and the distance to the external object.

[0159] In operation (740), the wearable device (103) may display the corrected image(s) so that they appear (or are recognized) to be located at a depth according to the corrected distance. For example, the wearable device (103) may display the corrected image(s) so that they appear (or are recognized) to be located at the depth according to the corrected distance, via at least one display of the wearable device (103) (e.g., display (510) of FIG. 5), according to operation (720), operation (725), or operation (735). For example, the corrected image(s) may be an image that is corrected or reprojected from the image(s) acquired in operation (705).

[0160] In operation (745), the wearable device (103) may display the generated image(s) so as to be positioned (or recognized) at a depth according to the distance to the external object. For example, if the distance to the external object is greater than or equal to a third reference distance, the wearable device (103) may display the generated image(s) so as to be positioned (or recognized) at a depth according to the distance to the external object acquired in operation (705) through the at least one display. In other words, the wearable device (103) may refrain from (or skip, delay, or exclude) performing correction of the image for an external object positioned in a remaining area different from the partial areas of the area.

[0161] Although not illustrated in FIG. 7, if the wearable device (103) determines that the distance to the external object is less than the first reference distance in operation (715), it may further determine whether the distance to the external object is included in a fourth sub-region. For example, the fourth sub-region may represent a region (e.g., about 50 cm to about 1 m) extending from a portion of the first sub-region toward the second sub-region to the first reference distance. For example, the fourth sub-region may be determined based on a minimum measurable distance of the RGB camera (e.g., about 50 cm) and a maximum measurable distance of the first distance sensor (e.g., about 1 m) (or the first reference distance). The wearable device (103) may identify a corrected distance based on the first distance information, the second distance information, and the distance to the external object when the distance to the external object is included within the fourth partial region (or, when the distance to the external object is greater than or equal to the fourth reference distance (e.g., the minimum measurable distance of the RGB camera) and less than the first reference distance). In this case, the first weight to be applied to the first distance information may be greater than the second weight to be applied to the second distance information. Alternatively, in operation (725), the first weight to be applied to the first distance information may be less than the second weight to be applied to the second distance information.

[0162] In the example of FIG. 7, an example is described of determining data (e.g., an image, first distance information, and second distance information) to be used to identify a corrected distance by comparing a distance to an external object with a plurality of reference distances, and identifying the corrected distance using the determined data, but the present disclosure is not limited thereto. For example, the wearable device (103) may also determine a ratio (or usage frequency) between data to be used by comparing a distance with a plurality of reference distances.

[0163] For example, the wearable device (103) may identify a corrected distance based on the distance to the external object, the first distance information, and the second distance information when the distance to the external object is less than the first reference distance, as in the example of operation (720). At this time, among the distance to the external object, the first distance information, and the second distance information, the value of the weight (or the first weight) for the first distance information may be the largest. A largest value of the first weight may indicate that the first distance information is used most frequently or is considered at the largest rate when calculating the corrected distance.

[0164] In addition, the wearable device (103) can identify a corrected distance based on the distance to the external object, the first distance information, and the second distance information when the distance to the external object is greater than or equal to the first reference distance and less than the second reference distance, as in the example of operation (725). Among the distance to the external object, the first distance information, and the second distance information, the value of the weight (or the third weight) for the distance to the external object may be the largest. The largest value of the third weight may indicate that the distance to the external object is used most frequently or is considered at the largest rate when calculating the corrected distance.

[0165] In addition, the wearable device (103) can identify a corrected distance based on the distance to the external object, the first distance information, and the second distance information when the distance to the external object is greater than or equal to the second reference distance and less than the third reference distance, as in the example of operation (735). Among the distance to the external object, the first distance information, and the second distance information, the weight (or second weight) for the second distance information may have the largest value. A largest value of the second weight may indicate that the second distance information is used most frequently or is considered at the largest rate when calculating the corrected distance.

[0166] The first reference distance, the second reference distance, and the third reference distance are exemplified as being determined based on the measurable distances of each of the components (e.g., the first distance sensor, the second distance sensor, and at least one camera (or image sensor)), but the present disclosure is not limited thereto. According to one embodiment, each of the first reference distance, the second reference distance, and the third reference distance may be adjusted (or changed) using the characteristics (or performance) of each sensor, information about the external environment (e.g., illuminance), or sensor information obtained from each sensor.

[0167] According to one embodiment, the wearable device (103) can identify the corrected distance by further utilizing additional information acquired through the first distance sensor, the second distance sensor, or the RGB camera. For example, the wearable device (103) can increase the accuracy of the corrected distance by utilizing additional information acquired through the first distance sensor, the second distance sensor, or the RGB camera in each of operation (720), operation (725), or operation (735). For example, the additional information can include sensor information indicating a distance to another external object acquired through the first distance sensor or the second distance sensor, or a distance to another external object acquired through image(s) acquired through the RGB camera.

[0168] Fig. 8a illustrates an example of a saliency map. Fig. 8a illustrates examples (800) of saliency maps (820, 840) obtained from images (810, 830) acquired through a camera of a wearable device (103) (e.g., camera (540) of Fig. 5). The saliency map may include images in which at least a portion of the images acquired through the camera (540) are visually emphasized (e.g., highlighted) according to the likelihood that a user's gaze will be located on the wearable device (103). For example, the likelihood that a gaze will be located may represent the probability that a user will focus and look at it. For example, the saliency map may be referred to as an interest map.

[0169] Referring to example (800), the wearable device (103) can acquire an image (810) and an image (830) through a camera (540). For example, the image (810) can include a visual object (811) corresponding to an external object (e.g., a puppy) within an external environment. For example, the image (830) can include visual objects (831, 832) corresponding to external objects (e.g., a vehicle) within an external environment.

[0170] According to one embodiment, the wearable device (103) may acquire saliency maps (820, 840) from images (810, 830) according to an image analysis technique. For example, the image analysis technique may include an estimation method using statistical information on the movement of an external object, statistical information of an image, or edge information on the boundary of an external object (or a visual object). Alternatively, for example, the wearable device (103) may acquire saliency maps (820, 840) through a camera (540).

[0171] Referring to example (800), the saliency map (820) may include a first portion (821) and a second portion (822) for visual objects (811). For example, the first portion (821) may represent a portion of the image (810) where a user's gaze is relatively likely to be located. For example, the second portion (822) may represent a portion of the image (810) where a user's gaze is relatively less likely to be located, compared to the first portion (821). For example, portions other than the second portion (822) may represent portions of the image (810) where a user's gaze is very unlikely to be located. In addition, the saliency map (840) may include a first portion (841) and a second portion (842) for visual objects (831, 832). For example, the first portion (841) may represent a portion of the image (830) where the user's gaze is relatively more likely to be located compared to the second portion (842). For example, the second portion (842) may represent a portion of the image (830) where the user's gaze is relatively less likely to be located compared to the first portion (841). For example, portions other than the second portion (842) may represent portions of the image (830) where the user's gaze is very unlikely to be located.

[0172] Referring to the above, the probability may gradually decrease as one moves from the boundary of the first portion (821 or 841) to the boundary of the second portion (822 or 842). The wearable device (103) may use the acquired saliency map (820 or 840) to calculate weights between data (e.g., an image acquired through the camera (540), first distance information acquired through the first distance sensor, and second distance information acquired through the second distance sensor) to be used for measuring the distance to an external object, and may identify a corrected distance by applying the calculated weights. Specific details related thereto may be referenced to FIGS. 8B and 8C below.

[0173] FIGS. 8B and 8C illustrate examples of a flow of operations for a method for a wearable device to identify a corrected distance using weights based on a saliency map.

[0174] At least some of the methods of FIGS. 8B and 8C may be performed by the wearable device (103) of FIG. 5. For example, at least some of the methods may be controlled by the processor (510) of the wearable device (103). In the embodiments described below, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0175] Referring to FIG. 8B, in operation (850), the wearable device (103) may acquire a saliency map. For example, the saliency map may be acquired from an image acquired through a camera (540), as described in example (800) of FIG. 8A. Alternatively, for example, the saliency map may also be acquired through the camera (540).

[0176] In operation (855), the wearable device (103) may identify a weighted region based on a distance and saliency map. For example, the distance may represent a distance to an external object acquired using image(s) acquired through the camera (540). For example, the weighted region may include a sub-region where the distance to the external object is located. For example, the sub-region may be determined based on a plurality of reference distances. For example, the weighted region may be included in a second sub-region (e.g., the second sub-region (412) of FIG. 4 ). However, the present disclosure is not limited thereto. For example, the weighted region may be included in a first sub-region (e.g., the first sub-region (411) of FIG. 4 ), a third sub-region (e.g., the third sub-region (413) of FIG. 4 ), or a fourth sub-region (e.g., the fourth sub-region (414) included in the first sub-region (411) of FIG. 4 ).

[0177] In operation (860), the wearable device (103) may calculate weights for a weight region. For example, the wearable device (103) may calculate a weight (or a third weight) for a distance to an external object, a weight (or a first weight) for first distance information (or a depth image) acquired through a first distance sensor (e.g., the first distance sensor (621) of FIG. 6B), and a weight (or a second weight) for second distance information (or a depth image) acquired through a second distance sensor (e.g., the second distance sensor (641) of FIG. 6C). For example, the wearable device (103) may calculate, for a weight region, a first weight for the first distance information, a second weight for the second distance information, and a third weight for the distance to an external object. However, the present disclosure is not limited thereto. For example, the wearable device (103) may calculate the first weight and the second weight, excluding the third weight.

[0178] In operation (865), the wearable device (103) can identify the corrected distance by applying weights to the distance information. For example, the wearable device (103) can identify the corrected distance based on the distance to the external object, the first distance information, and the second distance information. At this time, the corrected distance can be identified by applying the first weight to the first distance information, the second weight to the second distance information, and the third weight to the distance to the external object. For example, the first weight, the second weight, and the third weight can be referenced as an operating ratio between the RGB camera and the distance sensors. Alternatively, for example, the corrected distance can be identified by applying the first weight to the first distance information and the second weight to the second distance information. This may vary depending on the method (e.g., interpolation method or normalization method) used to identify the corrected distance.

[0179] Referring to FIG. 8c, in operation (870), the wearable device (103) may obtain a saliency map. For example, the saliency map may be obtained from an image obtained through a camera (540), as described in example (800) of FIG. 8a.

[0180] In operation (875), the wearable device (103) may obtain gaze region information. For example, the wearable device (103) may obtain gaze region information indicating an area (or gaze region) where the user's gaze is located through a gaze tracking camera. For example, the gaze tracking camera may be an example of the gaze tracking camera (260-1) of FIG. 2b or the cameras (260-1) of FIG. 3a. In other words, unlike a salient map, the gaze region information may indicate an area where the user's gaze is actually located.

[0181] In operation (880), the wearable device (103) may identify a weighted region based on the gaze region information and the saliency map. For example, the weighted region may include a sub-region representing an area where the gaze region information overlaps with a first part of the saliency map (e.g., the first part (821 or 842) of FIG. 8A). For example, the sub-region may be determined based on a plurality of reference distances. For example, the weighted region may be included in a first sub-region (e.g., the first sub-region (411) of FIG. 4), a third sub-region (e.g., the third sub-region (413) of FIG. 4), or a fourth sub-region (e.g., the fourth sub-region (414) included in the first sub-region (411) of FIG. 4).

[0182] In operation (885), the wearable device (103) may calculate weights for a weight region. For example, the wearable device (103) may calculate a weight (or a third weight) for a distance to an external object, a weight (or a first weight) for first distance information (or a depth image) acquired through a first distance sensor (e.g., the first distance sensor (621) of FIG. 6B), and a weight (or a third weight) for second distance information (or a depth image) acquired through a second distance sensor (e.g., the second distance sensor (641) of FIG. 6C). For example, the wearable device (103) may calculate at least one weight for a weight region. For example, the wearable device (103) may calculate a first weight for the first distance information, a second weight for the second distance information, and / or a third weight for the distance to an external object. However, the present disclosure is not limited thereto. According to one embodiment, the wearable device (103) may calculate at least one weight. For example, the wearable device (103) may calculate a first weight and a second weight, excluding a third weight.

[0183] In operation (890), the wearable device (103) can identify the corrected distance by applying the weights to the distance information. For example, the wearable device (103) can identify the corrected distance based on the distance to the external object, the first distance information, and the second distance information. At this time, the corrected distance can be identified by applying a first weight to the first distance information, a second weight to the second distance information, and a third weight to the distance to the external object. For example, the first weight, the second weight, and the third weight can be referenced as an operating ratio between the RGB camera and the distance sensors. Alternatively, for example, the corrected distance can be identified by applying the first weight to the first distance information and the second weight to the second distance information. This may vary depending on the method (e.g., interpolation method or normalization method) used to identify the corrected distance.

[0184] Figure 9 illustrates an example of a speed at which distance is measured by at least one sensor. The at least one sensor may include an RGB camera, an indirect-time of flight (I-ToF) sensor, and a direct-ToF (D-ToF) sensor. However, the present disclosure is not limited to the sensors described above.

[0185] FIG. 9 illustrates an example of a speed at which a wearable device (103) acquires data for measuring a distance to an external object using an RGB camera, an I-ToF sensor, and a D-ToF sensor. According to one embodiment, the speed at which the wearable device acquires data may be referred to as a distance measurement speed or a data acquisition cycle. The RGB camera may be an example of the camera (540) of FIG. 5 or the cameras (601, 602) of FIG. 6A. The I-ToF sensor may be an example of the sensor (530) of FIG. 5 or the first distance sensor (621) of FIG. 6B. The D-ToF sensor may be an example of the sensor (530) of FIG. 5 or the second distance sensor (641) of FIG. 6C.

[0186] Referring to FIG. 9, a first example (910) in which a wearable device (103) acquires an image through an RGB camera, a second example (920) in which a wearable device (103) acquires first distance information (or depth image) through an I-ToF sensor, and a third example (930) in which a wearable device (103) acquires second distance information (or depth image) through a D-ToF sensor are illustrated.

[0187] In one embodiment, referring to the first example (910), the wearable device (103) may acquire images through the RGB camera at a cycle having a first time interval. For example, the wearable device (103) may acquire images from each of the frames (911, 912, 913, 914). For example, the first time interval may represent the time length between the frames (911) and (912).

[0188] According to one embodiment, referring to the second example (920), the wearable device (103) can acquire first distance information (or depth image) according to a cycle having a second time interval through the I-ToF sensor. For example, the wearable device (103) can acquire first distance information from each of the frames (921, 922, 923). For example, the second time interval can represent the time length between the frame (921) and the frame (922). For example, the second time interval can be longer than the first time interval.

[0189] According to one embodiment, referring to the third example (930), the wearable device (103) can acquire second distance information (or depth image) according to a cycle having a third time interval through the D-ToF sensor. For example, the wearable device (103) can acquire second distance information from each of the frames (931, 932). For example, the third time interval can represent the time length between the frame (931) and the frame (932). For example, the third time interval can be longer than the second time interval.

[0190] At timings corresponding to frames (911, 921, 931), the wearable device (103) can acquire an image, first distance information, and second distance information. The wearable device (103) can identify a distance using the image acquired in frame (911). The wearable device (103) can identify a corrected distance based on at least one of the distance to an external object, the first distance information, and the second distance information. For example, when the distance to an external object is less than a first reference distance, the wearable device (103) can identify a corrected distance based on the distance to an external object and the first distance information. For example, when the distance to an external object is greater than or equal to the first reference distance and less than the second reference distance, the wearable device (103) can identify a corrected distance based on the distance to an external object, the first distance information, and the second distance information. For example, the wearable device (103) can identify a corrected distance based on the distance to the external object and the second distance information when the distance to the external object is greater than or equal to the second reference distance and less than the third reference distance. For example, the wearable device (103) can use the distance to the external object when the distance to the external object is less than the third reference distance. For example, the wearable device (103) can identify the corrected distance using an interpolation method at timings corresponding to frames (911, 921, 931).

[0191] At a timing corresponding to frame (912), the wearable device (103) can acquire an image. The wearable device (103) can identify a distance using the image acquired in frame (912). The wearable device (103) can identify a corrected distance based on at least one of the distance to an external object, the first distance information acquired in frame (921), and the second distance information acquired in frame (931). For example, when the distance to an external object is less than a first reference distance, the wearable device (103) can identify a corrected distance based on the distance to an external object and the first distance information. For example, when the distance to an external object is greater than or equal to the first reference distance and less than the second reference distance, the wearable device (103) can identify a corrected distance based on the distance to an external object, the first distance information, and the second distance information. For example, the wearable device (103) can identify a corrected distance based on the distance to the external object and the second distance information when the distance to the external object is greater than or equal to the second reference distance and less than the third reference distance. For example, the wearable device (103) can use the distance to the external object when the distance to the external object is less than the third reference distance. For example, the wearable device (103) can identify the corrected distance using a normalization method at a timing corresponding to the frame (912).

[0192] At timings corresponding to frames (913) and (922), the wearable device (103) can acquire an image and first distance information. The wearable device (103) can identify a distance using the image acquired in frame (913). The wearable device (103) can identify a corrected distance based on at least one of the distance to an external object, the first distance information acquired in frame (922), and the second distance information acquired in frame (931). For example, when the distance to an external object is less than a first reference distance, the wearable device (103) can identify a corrected distance based on the distance to an external object and the first distance information. For example, when the distance to an external object is greater than or equal to the first reference distance and less than the second reference distance, the wearable device (103) can identify a corrected distance based on the distance to an external object, the first distance information, and the second distance information. For example, the wearable device (103) can identify a corrected distance based on the distance to the external object and the second distance information when the distance to the external object is greater than or equal to the second reference distance and less than the third reference distance. For example, the wearable device (103) can use the distance to the external object when the distance to the external object is less than the third reference distance. For example, the wearable device (103) can identify a corrected distance using a normalization method at the timing corresponding to frame (913) and frame (922). Therefore, the frames from each sensor do not need to be precisely synchronized, and the most recently available frames that are most temporally consistent can be selected from each sensor to perform distance correction.

[0193] FIG. 10 illustrates an example of an operational flow for a method in which a wearable device uses multiple sensors and at least one camera to identify a calibrated distance to an external object and display an image.

[0194] At least some of the methods of FIG. 10 may be performed by the wearable device (103) of FIG. 5. For example, at least some of the methods may be controlled by the processor (510) of the wearable device (103). In the embodiments described below, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0195] In operation (1010), the wearable device (103) can identify a distance from a reference position to an external object within the external environment using images of the external environment acquired through a camera system. For example, the camera system of the wearable device (103) may be included in the camera (540) of FIG. 5 . For example, the images may represent visual information representing an area of ​​the external environment acquired through at least one camera of the camera system. For example, the images may include a visual object corresponding to an external object. For example, a period of acquiring images through at least one camera may have a first time interval (e.g., a frame interval of 90 fps).

[0196] According to one embodiment, the wearable device (103) can obtain a distance to an external object using the acquired image. For example, the distance to the external object may be referred to as a temporary distance or an initial distance. For example, if at least one camera includes multiple RGB cameras, the wearable device (103) can obtain the distance to the external object based on the parallax between images acquired from the multiple RGB cameras. The reference position may include the position of at least one camera disposed in the wearable device (103). For example, the reference position may be determined based on a portion where at least one lens of at least one camera is positioned within the wearable device (103). For example, if at least one camera includes multiple cameras, the reference position may be positioned between multiple lenses of the multiple cameras. For example, the reference position may correspond to the position of a sensor portion (or plane, image plane) within the camera.

[0197] According to one embodiment, the wearable device (103) may determine a sub-region of the external environment in which a distance to an external object is located. For example, the sub-region may be included in the region. For example, the region may include a plurality of sub-regions. For example, the plurality of sub-regions may include a first sub-region, a second sub-region, and a third sub-region. For example, the first sub-region may represent an area between a reference location and a first reference distance. For example, the first sub-region may be referred to as a near-distance region. For example, the second sub-region may represent an area between a first reference distance and a second reference distance. For example, the second sub-region may be referred to as an intermediate-distance region. For example, the third sub-region may represent an area between a second reference distance and a third reference distance. For example, the third sub-region may be referred to as a far-distance region. For example, a subregion may be defined by a range of distances to an external object between a first reference value (e.g., reference position, first reference distance, second reference distance) and a second reference value (e.g., first reference distance, second reference distance, third reference distance).

[0198] For example, the first reference distance may be determined based on a distance measurable by the first distance sensor of the wearable device (103) (e.g., the first distance sensor (621) of FIG. 6B ). For example, the measurable distance of the first distance sensor may be from about 10 cm to about 1 m. The above example is merely an example for convenience of explanation, and the present disclosure is not limited thereto. For example, the first reference distance may be set to the maximum measurable distance of the first distance sensor (e.g., about 1 m).

[0199] For example, the second reference distance may be determined based on a distance measurable by the RGB camera of the wearable device (103). For example, the measurable distance of the RGB camera may be from about 50 cm to about 1.5 m. The above example is merely an example for convenience of explanation, and the present disclosure is not limited thereto. For example, the second reference distance may be set to a maximum measurable distance of at least one camera (e.g., about 1.5 m). For example, the second reference distance may have a value greater than the first reference distance.

[0200] For example, the third reference distance may be determined based on a distance measurable by the second distance sensor of the wearable device (103). For example, the measurable distance of the second distance sensor may be from about 20 cm to about 5 m. The above example is merely an example for convenience of explanation, and the present disclosure is not limited thereto. For example, the third reference distance may be set to the maximum measurable distance of the second distance sensor (e.g., about 5 m). For example, the third reference distance may have a value greater than the second reference distance.

[0201] The measurable distance may represent a measurement range having an accuracy higher than the reference accuracy of the RGB camera, the first distance sensor, and the second distance sensor. In other words, the wearable device (103) is not incapable of measuring a distance to an external object located beyond the measurable distance using at least one camera, the first distance sensor, and the second distance sensor. The accuracy (or reliability) of the measured distance to an external object located beyond the measurable distance may be lower than the reference accuracy.

[0202] In operation (1020), the wearable device (103) can identify a corrected distance from a reference position to an external object based on first distance information of the external object acquired through a first distance sensor, second distance information of the external object acquired through a second distance sensor, and the distance to the external object, when the distance to the external object is greater than or equal to a first reference distance and less than a second reference distance exceeding the first reference distance.

[0203] According to one embodiment, when the distance to the external object is less than the first reference distance, the wearable device (103) can identify a corrected distance from the reference position to the external object based on the first distance information of the external object obtained through the first distance sensor and the distance to the external object.

[0204] According to one embodiment, the wearable device (103) may determine whether the distance to the external object is less than the second reference distance. If the distance to the external object is less than the second reference distance, the wearable device (103) may determine whether the distance to the external object is less than the first reference distance. Conversely, if the distance to the external object is greater than or equal to the second reference distance, the wearable device (103) may determine whether the distance to the external object is less than a third reference distance.

[0205] For example, the wearable device (103) may identify a corrected distance based on the first distance information and the distance to the external object when the distance to the external object is less than the first reference distance. For example, the first distance information may include a depth image acquired through the first distance sensor. For example, the period in which the first distance information acquired through the first distance sensor is acquired may have a second time interval (e.g., a frame interval of 30 fps) that is longer than the first time interval.

[0206] For example, the wearable device (103) may identify a corrected distance based on the first distance information, the second distance information, and the distance to the external object when the distance to the external object is greater than or equal to the first reference distance and less than the second reference distance. For example, the second distance information may include a depth image acquired through a second distance sensor. For example, the period in which the second distance information acquired through the second distance sensor is acquired may have a third time interval (e.g., a frame interval of 10 fps) that is longer than the second time interval.

[0207] For example, the wearable device (103) can identify a corrected distance based on the second distance information and the distance to the external object when the distance to the external object is greater than or equal to the second reference distance and less than the third reference distance.

[0208] In operation (1030), the wearable device (103) may display a modified (or corrected) image(s) such that a visual object image (or visual object) corresponding to an external object appears (or is perceived) to be located at a depth according to a corrected distance. For example, the wearable device (103) may display the modified image(s) via a display system including at least one display. For example, the modified image may be an image corrected or reprojected from an image. The corrected or reprojected image may be adjusted for differences in viewing distances between cameras of the wearable device (103) and the eyes of a user wearing the wearable device (103), so that the user may perceive the appearance of the external object as being located at a natural viewing depth without an offset that may occur due to the thickness of the wearable device (103) and the camera position relative to the eyes of the user.

[0209] As another example, the wearable device (103) may display, through at least one display, image(s) generated so as to be positioned (or recognized) at a depth according to the distance to the external object when the distance to the external object is greater than or equal to a third reference distance. In other words, the wearable device (103) may refrain from (or skip, delay, exclude) performing correction of images for external objects positioned in the remaining areas that are different from the partial areas among the above areas.

[0210] Although not illustrated in FIG. 10, if the wearable device (103) determines that the distance to the external object is less than the first reference distance, the wearable device (103) may further determine whether the distance to the external object is included within a fourth sub-region. For example, the fourth sub-region may represent a region (e.g., about 50 cm to about 1 m) extending from a portion of the first sub-region toward the second sub-region to the first reference distance. For example, the fourth sub-region may be determined based on a minimum measurable distance of the RGB camera (e.g., about 50 cm) and a maximum measurable distance of the first distance sensor (e.g., about 1 m) (or the first reference distance). If the distance to the external object is included within the fourth sub-region (or, if the distance to the external object is greater than or equal to the fourth reference distance (e.g., the minimum measurable distance of the RGB camera) and less than the first reference distance), the wearable device (103) may also identify a corrected distance based on the first distance information, the second distance information, and the distance to the external object. In this case, the first weight to be applied to the first distance information may be greater than the second weight to be applied to the second distance information. Conversely, in operation (725), the first weight to be applied to the first distance information may be less than the second weight to be applied to the second distance information.

[0211] Although not illustrated in FIG. 10, in one embodiment, the wearable device (103) may identify the corrected distance further based on a saliency map. For example, the wearable device (103) may calculate weights to be applied to the distance to an external object acquired using an image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor, based on the saliency map including information about the user's area of ​​interest. For example, the area of ​​interest may be referred to as a weighted area. For example, the wearable device (103) may adjust the operating ratio between at least one camera and sensors by applying the calculated weights to the distance to an external object, the first distance information, and the second distance information. Furthermore, for example, the wearable device (103) may further use the area toward which the user's gaze is directed (or gaze range) in addition to the saliency map to identify the area of ​​interest.

[0212] According to one embodiment, the wearable device (103) may identify a weighted region based on gaze region information and a saliency map. For example, the weighted region may include a sub-region representing an area where the gaze region information overlaps with a first portion of the saliency map (e.g., the first portion (821 or 842) of FIG. 8A). For example, the sub-region may be determined based on a plurality of reference distances. For example, the weighted region may be included in a first sub-region (e.g., the first sub-region (411) of FIG. 4), a third sub-region (e.g., the third sub-region (413) of FIG. 4), or a fourth sub-region (e.g., the fourth sub-region (414) included in the first sub-region (411) of FIG. 4).

[0213] According to one embodiment, the wearable device (103) may calculate weights for a weight region. For example, the wearable device (103) may calculate a weight (or a third weight) for a distance to an external object, a weight (or a first weight) for first distance information (or a depth image) acquired through a first distance sensor (e.g., the first distance sensor (621) of FIG. 6B), and a weight (or a second weight) for second distance information (or a depth image) acquired through a second distance sensor (e.g., the second distance sensor (641) of FIG. 6C). For example, the wearable device (103) may calculate at least one weight for a weight region. For example, the wearable device (103) may calculate a first weight for the first distance information, a second weight for the second distance information, and / or a third weight for the distance to an external object. However, the present disclosure is not limited thereto. In one embodiment, the wearable device (103) can calculate at least one weight. For example, the wearable device (103) can calculate the first weight and the second weight, excluding the third weight. In one embodiment, the wearable device (103) can identify the corrected distance by applying the weight to the distance information.

[0214] According to one embodiment, the wearable device (103) can identify a corrected distance based on scene analysis of an image acquired through at least one camera. For example, the scene analysis can include analysis based on SLAM. For example, the wearable device (103) can identify the location of each of one or more external objects within the image based on the scene analysis. For example, the wearable device (103) can store information about the location of each of one or more external objects as scene understanding data. For example, the scene understanding data can be used in a software application executed by the wearable device (103). For example, the scene understanding data can be used for a service provided by the software application. When the scene understanding data is stored in a number greater than or equal to a reference data number, the wearable device (103) can recognize that one or more objects of interest are located within a specific sub-region among the sub-regions. For example, the wearable device (103) may identify a specific sub-region where one or more objects of interest are located as a region of interest (or weighted region). For example, the object of interest may include an external object of interest to the user. Accordingly, the wearable device (103) may calculate weights to be applied to the distance to the external object acquired through the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor. For example, the wearable device (103) may adjust the operating ratio between the sensors by applying the calculated weights to the distance to the external object, the first distance information, and the second distance information.

[0215] According to one embodiment, the wearable device (103) can identify the corrected distance based on the user's gesture. For example, the wearable device (103) can identify a gesture or interaction by the user's hand through a sensor or a camera. When the gesture or interaction is identified, the wearable device (103) can identify a first sub-region (e.g., the first sub-region (411) of FIG. 4) as a region of interest, and accordingly calculate a weight to be applied to the distance to an external object acquired through the image and the first distance information of the first distance sensor. However, the present disclosure is not limited thereto. For example, when the wearable device (103) identifies a gesture or interaction, the wearable device (103) can identify a fourth sub-region (e.g., the fourth sub-region (414) of FIG. 4) within the first sub-region as a region of interest, and calculate a weight to be applied to the distance to an external object acquired through the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor. Alternatively, for example, if the wearable device (103) identifies a gesture or interaction, it may identify a first partial region as a region of interest and calculate weights to be applied to the distance to an external object acquired through the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor. In this case, for higher accuracy even within the first partial region, the wearable device (103) may identify a corrected distance to an external object using the second distance information together with the first distance information.

[0216] According to one embodiment, when the illuminance of the external environment is lower than the reference illuminance, the wearable device (103) may identify the distance to an external object in the external environment based on second distance information of the second distance sensor instead of an image acquired from at least one camera. By comparing the distance to the external object identified based on the second distance information with the partial regions, the wearable device (103) may determine the distance information to be used.

[0217] According to one embodiment, when the surface of the external object is made of a material that reflects light, the wearable device (103) may calculate a weight to be applied to an image acquired through at least one camera (or a distance to the external object acquired using the image) as a relatively higher value compared to the distance information acquired from the first distance sensor and the second distance sensor. In addition, when processing an area in the external environment where light is scattered (e.g., a periphery (or corner) of the external object), the wearable device (103) may calculate a weight to be applied to an image acquired through at least one camera (or a distance acquired using the image) as a relatively higher value compared to the distance information acquired from the first distance sensor and the second distance sensor.

[0218] FIG. 11 illustrates an example of an operational flow for a method of identifying a corrected distance to an external object and displaying an image by comparing a distance to an external object acquired by a wearable device using at least one camera with a plurality of criteria.

[0219] At least some of the methods of FIG. 11 may be performed by the wearable device (103) of FIG. 5. For example, at least some of the methods may be controlled by the processor (510) of the wearable device (103). In the embodiments described below, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0220] In operation (1110), the wearable device (103) can identify a distance from a reference position to an external object within the external environment using images of the external environment acquired through a camera system. For example, the camera system of the wearable device (103) may be included in the camera (540) of FIG. 5 . For example, the images may represent visual information representing an area of ​​the external environment acquired through at least one camera of the camera system. For example, the images may include a visual object corresponding to an external object. For example, the period of acquiring images through at least one camera may have a first time interval (e.g., a frame interval of 90 fps).

[0221] According to one embodiment, the wearable device (103) can obtain a distance to an external object using the acquired image. For example, the distance to the external object may be referred to as a temporary distance or an initial distance. For example, if at least one camera includes multiple RGB cameras, the wearable device (103) can obtain the distance to the external object based on the parallax between images acquired from the multiple RGB cameras. The reference position may include the position of at least one camera disposed in the wearable device (103). For example, the reference position may be determined based on a portion where at least one lens of at least one camera is positioned within the wearable device (103). For example, if at least one camera includes multiple cameras, the reference position may be positioned between multiple lenses of the multiple cameras. For example, the reference position may correspond to the position of a sensor portion (or plane, image plane) within the camera.

[0222] According to one embodiment, the wearable device (103) may determine a sub-region of the external environment in which a distance to an external object is located. For example, the sub-region may be included in the region. For example, the region may include a plurality of sub-regions. For example, the plurality of sub-regions may include a first sub-region, a second sub-region, and a third sub-region. For example, the first sub-region may represent an area between a reference location and a first reference distance. For example, the first sub-region may be referred to as a near-distance region. For example, the second sub-region may represent an area between a first reference distance and a second reference distance. For example, the second sub-region may be referred to as an intermediate-distance region. For example, the third sub-region may represent an area between a second reference distance and a third reference distance. For example, the third sub-region may be referred to as a far-distance region. For example, a subregion may be defined by a range of distances to an external object between a first reference value (e.g., reference position, first reference distance, second reference distance) and a second reference value (e.g., first reference distance, second reference distance, third reference distance).

[0223] For example, the first reference distance may be determined based on a distance measurable by the first distance sensor of the wearable device (103) (e.g., the first distance sensor (621) of FIG. 6B ). For example, the measurable distance of the first distance sensor may be from about 10 cm to about 1 m. The above example is merely an example for convenience of explanation, and the present disclosure is not limited thereto. For example, the first reference distance may be set to the maximum measurable distance of the first distance sensor (e.g., about 1 m).

[0224] For example, the second reference distance may be determined based on a distance measurable by the RGB camera of the wearable device (103). For example, the measurable distance of the RGB camera may be from about 50 cm to about 1.5 m. The above example is merely an example for convenience of explanation, and the present disclosure is not limited thereto. For example, the second reference distance may be set to a maximum measurable distance of at least one camera (e.g., about 1.5 m). For example, the second reference distance may have a value greater than the first reference distance.

[0225] For example, the third reference distance may be determined based on a distance measurable by the second distance sensor of the wearable device (103). For example, the measurable distance of the second distance sensor may be from about 20 cm to about 5 m. The above example is merely an example for convenience of explanation, and the present disclosure is not limited thereto. For example, the third reference distance may be set to the maximum measurable distance of the second distance sensor (e.g., about 5 m). For example, the third reference distance may have a value greater than the second reference distance.

[0226] The measurable distance may represent a measurement range having an accuracy higher than the reference accuracy of the RGB camera, the first distance sensor, and the second distance sensor. In other words, the wearable device (103) is not incapable of measuring a distance to an external object located beyond the measurable distance using at least one camera, the first distance sensor, and the second distance sensor. The accuracy (or reliability) of the measured distance to an external object located beyond the measurable distance may be lower than the reference accuracy.

[0227] In operation (1120), the wearable device (103) may compare the distance to the external object with a plurality of criteria. The criteria may be related to measurable distances (e.g., distance ranges) of the sensors. For example, the plurality of criteria may include a first criterion and a second criterion. For example, the first criterion may include a condition that the distance to the external object is less than the first criterion distance. The second criterion may include a condition that the distance to the external object is greater than or equal to the first criterion distance and less than the second criterion distance that exceeds the first criterion distance. In operation (1130), the wearable device (103) may identify the distance to the external object and the corrected distance using at least one sensor information of the sensors. For example, the wearable device (103) may identify the distance to the external object and the corrected distance using at least one sensor information based on the result of the comparison.

[0228] According to one embodiment, the wearable device (103) can identify a corrected distance from a reference position to an external object based on first distance information of the external object obtained through the first distance sensor and the distance to the external object when the first criterion is satisfied.

[0229] According to one embodiment, the wearable device (103) can identify a corrected distance from a reference position to an external object based on first distance information of the external object acquired through the first distance sensor, second distance information of the external object acquired through the second distance sensor, and the distance to the external object when the second criterion is satisfied.

[0230] For example, the first distance information may include a depth image acquired through the first distance sensor. For example, the period in which the first distance information acquired through the first distance sensor is acquired may have a second time interval (e.g., a frame interval of 30 fps) that is longer than the first time interval. For example, the second distance information may include a depth image acquired through the second distance sensor. For example, the period in which the second distance information acquired through the second distance sensor is acquired may have a third time interval (e.g., a frame interval of 10 fps) that is longer than the second time interval.

[0231] In the above example, the multiple criteria include a first criterion and a second criterion, but the present disclosure is not limited thereto. For example, the multiple criteria may include three or more criteria.

[0232] According to one embodiment, the wearable device (103) may identify a corrected distance based on the second distance information and the distance to the external object if a third condition is satisfied that the distance to the external object exceeds the second reference distance and is less than the third reference distance. Alternatively, the wearable device (103) may identify a corrected distance based on the first distance information, the second distance information, and the distance to the external object if a fourth condition is satisfied that the distance to the external object is less than the first reference distance and exceeds the fourth reference distance that is less than the first reference distance. For example, the wearable device (103) may further determine whether the distance to the external object is included within a fourth sub-region. For example, the fourth sub-region may represent a region (e.g., about 50 cm to about 1 m) extending from a portion of the first sub-region toward the second sub-region to the first reference distance. For example, the fourth sub-area may be determined based on a minimum measurable distance of the RGB camera (e.g., about 50 cm) and a maximum measurable distance of the first distance sensor (e.g., about 1 m) (or a first reference distance). The wearable device (103) may also identify a corrected distance based on the first distance information, the second distance information, and the distance to the external object when the distance to the external object is included in the fourth sub-area (or when the distance to the external object is greater than or equal to the fourth reference distance (e.g., the minimum measurable distance of the RGB camera) and less than the first reference distance). In this case, the first weight to be applied to the first distance information may be greater than the second weight to be applied to the second distance information. Conversely, when the second condition is satisfied, the first weight to be applied to the first distance information may be less than the second weight to be applied to the second distance information.

[0233] In operation (1140), the wearable device (103) may display one or more images that have been modified (or corrected) so that a visual object image (or visual object) corresponding to an external object appears (or is perceived) to be located at a depth according to a corrected distance. For example, the wearable device (103) may display the modified image through a display system including at least one display. For example, the modified image may be an image that has been corrected or reprojected from an image, and the modified images may then continue to make depth adjustments as the user of the wearable device (103) moves or the external object moves.

[0234] For example, the wearable device (103) may display, through at least one display, image(s) generated so as to be positioned at a depth according to the distance to the external object (or recognized) when the fifth condition that the distance to the external object is greater than or equal to the third reference distance is satisfied. In other words, the wearable device (103) may refrain from (or skip, delay, exclude) performing correction of images for external objects positioned in the remaining areas that are different from the partial areas among the above areas.

[0235] Although not illustrated in FIG. 11, in one embodiment, the wearable device (103) may identify the corrected distance further based on a saliency map. For example, the wearable device (103) may calculate weights to be applied to the distance to an external object acquired using an image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor, based on the saliency map including information about the user's area of ​​interest. For example, the area of ​​interest may be referred to as a weighted area. For example, the wearable device (103) may adjust the operating ratio between at least one camera and sensors by applying the calculated weights to the distance to an external object, the first distance information, and the second distance information. Furthermore, for example, the wearable device (103) may further use the area toward which the user's gaze is directed (or gaze range) in addition to the saliency map to identify the area of ​​interest.

[0236] According to one embodiment, the wearable device (103) may identify a weighted region based on gaze region information and a saliency map. For example, the weighted region may include a sub-region representing an area where the gaze region information overlaps with a first portion of the saliency map (e.g., the first portion (821 or 842) of FIG. 8A). For example, the sub-region may be determined based on a plurality of reference distances. For example, the weighted region may be included in a first sub-region (e.g., the first sub-region (411) of FIG. 4), a third sub-region (e.g., the third sub-region (413) of FIG. 4), or a fourth sub-region (e.g., the fourth sub-region (414) included in the first sub-region (411) of FIG. 4).

[0237] According to one embodiment, the wearable device (103) may calculate weights for a weight region. For example, the wearable device (103) may calculate a weight (or a third weight) for a distance to an external object, a weight (or a first weight) for first distance information (or a depth image) acquired through a first distance sensor (e.g., the first distance sensor (621) of FIG. 6B), and a weight (or a third weight) for second distance information (or a depth image) acquired through a second distance sensor (e.g., the second distance sensor (641) of FIG. 6C). For example, the wearable device (103) may calculate at least one weight for a weight region. For example, the wearable device (103) may calculate a first weight for the first distance information, a second weight for the second distance information, and / or a third weight for the distance to an external object. However, the present disclosure is not limited thereto. In one embodiment, the wearable device (103) can calculate at least one weight. For example, the wearable device (103) can calculate the first weight and the second weight, excluding the third weight. In one embodiment, the wearable device (103) can identify the corrected distance by applying the weight to the distance information.

[0238] According to one embodiment, the wearable device (103) can identify a corrected distance based on scene analysis of an image acquired through at least one camera. For example, the scene analysis can include analysis based on SLAM. For example, the wearable device (103) can identify the location of each of one or more external objects within the image based on the scene analysis. For example, the wearable device (103) can store information about the location of each of one or more external objects as scene understanding data. For example, the scene understanding data can be used in a software application executed by the wearable device (103). For example, the scene understanding data can be used for a service provided by the software application. When the scene understanding data is stored in a number greater than or equal to a reference data number, the wearable device (103) can recognize that one or more objects of interest are located within a specific sub-region among the sub-regions. For example, the wearable device (103) may identify a specific sub-region where one or more objects of interest are located as a region of interest (or weighted region). For example, the object of interest may include an external object of interest to the user. Accordingly, the wearable device (103) may calculate weights to be applied to the distance to the external object acquired through the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor. For example, the wearable device (103) may adjust the operating ratio between the sensors by applying the calculated weights to the distance to the external object, the first distance information, and the second distance information.

[0239] According to one embodiment, the wearable device (103) can identify the corrected distance based on the user's gesture. For example, the wearable device (103) can identify a gesture or interaction by the user's hand through a sensor or a camera. When the gesture or interaction is identified, the wearable device (103) can identify a first sub-region (e.g., the first sub-region (411) of FIG. 4) as a region of interest, and accordingly calculate a weight to be applied to the distance to an external object acquired through the image and the first distance information of the first distance sensor. However, the present disclosure is not limited thereto. For example, when the wearable device (103) identifies a gesture or interaction, the wearable device (103) can identify a fourth sub-region (e.g., the fourth sub-region (414) of FIG. 4) within the first sub-region as a region of interest, and calculate a weight to be applied to the distance to an external object acquired through the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor. Alternatively, for example, if the wearable device (103) identifies a gesture or interaction, it may identify a first partial region as a region of interest and calculate weights to be applied to the distance to an external object acquired through the image, the first distance information of the first distance sensor, and the second distance information of the second distance sensor. In this case, for higher accuracy even within the first partial region, the wearable device (103) may identify a corrected distance to an external object using the second distance information together with the first distance information.

[0240] According to one embodiment, when the illuminance of the external environment is lower than the reference illuminance, the wearable device (103) may identify the distance to an external object in the external environment based on second distance information of the second distance sensor instead of an image acquired from at least one camera. By comparing the distance to the external object identified based on the second distance information with the partial regions, the wearable device (103) may determine the distance information to be used.

[0241] According to one embodiment, when the surface of the external object is made of a material that reflects light, the wearable device (103) may calculate a weight to be applied to an image acquired through at least one camera (or a distance to the external object acquired using the image) as a relatively higher value compared to the distance information acquired from the first distance sensor and the second distance sensor. In addition, when processing an area in the external environment where light is scattered (e.g., a periphery (or corner) of the external object), the wearable device (103) may calculate a weight to be applied to an image acquired through at least one camera (or a distance acquired using the image) as a relatively higher value compared to the distance information acquired from the first distance sensor and the second distance sensor.

[0242] In the above examples of FIGS. 1 to 11, the present disclosure is described as being applied to a wearable device (103) providing VST, but the present disclosure is not limited thereto. The present disclosure can also be applied to a wearable device (103) providing an AR environment. For example, when displaying a visual object to be added to an external object, the wearable device (103) may identify a distance and a corrected distance to the external object, and display the visual object according to the corrected distance. For example, the visual object may be displayed by at least partially overlapping the external object or being positioned in a surrounding area of ​​the external object. The wearable device (103) may display the visual object (or an image including the visual object, visual information) so that the visual object appears (or is recognized) to be positioned at a depth according to the corrected distance.

[0243] As described above, the device, method, and storage medium according to the present disclosure can utilize data acquired using a plurality of sensors (e.g., an I-ToF sensor and a D-ToF sensor) and at least one camera (e.g., at least one RGB camera) included in the wearable device (103). For example, the device, method, and storage medium according to the present disclosure can select data to be used for correcting a distance to an external object according to an area where the external object is located within an external environment (or a distance to the external object). The device, method, and storage medium according to the present disclosure can identify a distance to an external object (or a depth of a visual object corresponding to the external object) using at least one of an image acquired through the at least one camera or distance information acquired through the plurality of sensors. The device, method, and storage medium according to the present disclosure can reduce power consumption due to driving the plurality of sensors by determining at least one sensor among the plurality of sensors to be utilized using the distance to the external object. The device, method, and storage medium according to the present disclosure can perform correction (or reprojection) on an image to be displayed using a more accurately identified (or corrected) distance. Accordingly, the device, method, and storage medium according to the present disclosure can improve the user experience (e.g., reduce the sense of incongruity) by accurately identifying the distance (or depth).

[0244] As described above, the wearable device (103) may include a memory (550) including one or more storage media for storing instructions. The wearable device (103) may include at least one processor (510) including a processing circuit. The wearable device (103) may include a plurality of sensors (530) including a first distance sensor and a second distance sensor. The wearable device (103) may include a camera system including a plurality of cameras (540). The wearable device (103) may include a display system including a first display and a second display. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to identify a distance from a reference location to an external object within the external environment using images of the external environment acquired through the camera system. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to identify, when the distance is greater than or equal to a first reference distance and less than a second reference distance that exceeds the first reference distance, first distance information of the external object acquired through the first distance sensor, second distance information of the external object acquired through the second distance sensor, and a corrected distance from the reference location to the external object based on the distance. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to display, through the display system, one or more images that are modified so that a visual object image corresponding to the external object appears to be located at a depth according to the corrected distance.

[0245] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to identify the corrected distance from the reference location to the external object based on the first distance information and the distance, if the distance is less than the first reference distance.

[0246] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to identify the corrected distance from the reference location to the external object based on the second distance information and the distance, if the distance is greater than or equal to the second reference distance and less than a third reference distance that exceeds the second reference distance.

[0247] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to display, through the display system, one or more images generated such that the visual object image appears to be located at the depth according to the distance, if the distance is greater than or equal to the third reference distance.

[0248] In one embodiment, the first reference distance may be determined based on a measurable distance by the first distance sensor. The second reference distance may be determined based on a measurable distance by the camera system. The third reference distance may be determined based on a measurable distance by the second distance sensor.

[0249] According to one embodiment, the measurable distance of each of the first distance sensor, the at least one camera (540), and the second distance sensor may include a distance at which an error between an estimated distance and an actual distance is less than a reference difference.

[0250] In one embodiment, the period of acquiring the images through the camera system may have a first time interval. The period of acquiring the first distance information through the first distance sensor may have a second time interval longer than the first time interval. The period of acquiring the second distance information through the second distance sensor may have a third time interval longer than the second time interval.

[0251] In one embodiment, if the distance is less than the first reference distance within a frame in which the first time interval overlaps with the second time interval, the corrected distance may be calculated based on the first distance information. If the distance is less than the first reference distance within another frame in which the first time interval does not overlap with the second time interval, the corrected distance may be adjusted from the distance using the first distance information.

[0252] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to identify a corrected distance from the reference location to the external object based on the first distance information, the second distance information, and the distance, if the distance is greater than or equal to a fourth reference distance that is less than the first reference distance and less than the first reference distance. If the distance is greater than or equal to the fourth reference distance and less than the first reference distance, the corrected distance may be identified by applying a first weight to the first distance information and a second weight to the second distance information. If the distance is greater than or equal to the first reference distance and less than the second reference distance, the corrected distance may be identified by applying a third weight to the first distance information and a fourth weight to the second distance information. The first weight may exceed the second weight. The third weight may be less than the fourth weight.

[0253] In one embodiment, the fourth reference distance may correspond to a minimum measurable distance to the external object using the images from the plurality of cameras of the camera system.

[0254] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to obtain a saliency map in which at least a portion of an image is visually emphasized according to a likelihood that a gaze of a user of the wearable device (103) is located. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to identify a weighted region based on the saliency map. The instructions, when individually or collectively executed by the at least one processor (510), may cause the wearable device (103) to determine a first weight for the first distance information and a second weight for the second distance information for the weighted region. The first weight and the second weight may be used to identify the corrected distance.

[0255] In one embodiment, the weighted region may be identified based on the saliency map and the distance, or based on the saliency map and a gaze region according to the line of sight.

[0256] According to one embodiment, the reference position may include the positions of the plurality of cameras arranged in the wearable device (103).

[0257] In one embodiment, each of the plurality of cameras may include a red-green-blue (RGB) camera. The first distance sensor may include an indirect-time of flight (I-ToF) sensor. The second distance sensor may include a direct-time of flight (D-ToF) sensor.

[0258] As described above, the method performed by the wearable device (103) may include an operation of identifying a distance from a reference position to an external object in the external environment using images of the external environment acquired through a camera system including a plurality of cameras of the wearable device (103). The method may include an operation of identifying first distance information of the external object acquired through a first distance sensor of the wearable device (103), second distance information of the external object acquired through a second distance sensor of the wearable device (103), and a corrected distance from the reference position to the external object based on the distance, when the distance is greater than or equal to a first reference distance and less than a second reference distance that exceeds the first reference distance. The method may include an operation of displaying one or more images that are modified so that a visual object image corresponding to the external object appears to be located at a depth according to the corrected distance.

[0259] According to one embodiment, the method may include an operation of identifying the corrected distance from the reference location to the external object based on the first distance information and the distance, if the distance is less than the first reference distance.

[0260] According to one embodiment, the method may include an operation of identifying the corrected distance from the reference position to the external object based on the second distance information and the distance, when the distance is greater than or equal to the second reference distance and less than a third reference distance that exceeds the second reference distance.

[0261] According to one embodiment, the method may include displaying one or more images generated such that the visual object image appears to be located at the depth according to the distance.

[0262] In one embodiment, the first reference distance may be determined based on a measurable distance by the first distance sensor. The second reference distance may be determined based on a measurable distance by the camera system. The third reference distance may be determined based on a measurable distance by the second distance sensor.

[0263] According to one embodiment, the measurable distance of each of the first distance sensor, the at least one camera (540), and the second distance sensor may include a distance at which an error between an estimated distance and an actual distance is less than a reference difference.

[0264] According to one embodiment, the period for acquiring the image through the at least one camera (540) may have a first time interval. The period for acquiring the first distance information through the first distance sensor may have a second time interval longer than the first time interval. The period for acquiring the second distance information through the second distance sensor may have a third time interval longer than the second time interval.

[0265] As described above, the non-transitory computer-readable storage medium can store one or more programs that, when individually or collectively executed by at least one processor (510) of a wearable device (103) including a plurality of sensors (530) including a first distance sensor and a second distance sensor, a camera system including a plurality of cameras (540), and a display system including a first display and a second display, cause the wearable device (103) to identify a distance from a reference position to an external object in the external environment using images of the external environment acquired through the camera system. The non-transitory computer-readable storage medium may store one or more programs that, when individually or collectively executed by the at least one processor (510), store instructions that cause the wearable device (103) to compare the distance to the external object and the measurable distances of the sensors with criteria related to the distance. The non-transitory computer-readable storage medium may store one or more programs that, when individually or collectively executed by the at least one processor (510), store instructions that cause the wearable device (103) to identify a corrected distance from the reference position to the external object using the distance and the sensor information of the sensors.The non-transitory computer-readable storage medium may store one or more programs that, when individually or collectively executed by the at least one processor (510), cause the wearable device (103) to display, through the display system, one or more images modified so that a visual object image corresponding to the external object appears to be located at a depth according to the corrected distance.

[0266] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0267] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0268] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0269] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0270] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0271] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (103), A memory (550) comprising one or more storage media for storing instructions; At least one processor (510) comprising a processing circuit; A plurality of sensors (530) including a first distance sensor and a second distance sensor; A camera system comprising a plurality of cameras (540); and A display system comprising a first display and a second display, The above instructions, when individually or collectively executed by the at least one processor (510), cause the wearable device (103) to: Using images of the external environment acquired through the above camera system, the distance from a reference position to an external object within the external environment is identified, If the distance is greater than or equal to a first reference distance and less than a second reference distance that exceeds the first reference distance, identifying the first distance information of the external object acquired through the first distance sensor, the second distance information of the external object acquired through the second distance sensor, and the corrected distance from the reference position to the external object based on the distance, and Causing the display system to display one or more images, wherein the visual object image corresponding to the external object is modified to appear positioned at a depth according to the corrected distance. Wearable device (103).

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (510), cause the wearable device (103) to: If the distance is less than the first reference distance, causing the corrected distance from the reference position to the external object to be identified based on the first distance information and the distance. Wearable device (103).

3. In claim 2, The above instructions, when individually or collectively executed by the at least one processor (510), cause the wearable device (103) to: If the distance is greater than or equal to the second reference distance and less than a third reference distance that exceeds the second reference distance, causing the corrected distance from the reference position to the external object to be identified based on the second distance information and the distance. Wearable device (103).

4. In claim 3, The above instructions, when individually or collectively executed by the at least one processor (510), cause the wearable device (103) to: If the distance is greater than or equal to the third reference distance, causing the display system to display one or more images generated so that the visual object image appears to be located at the depth according to the distance. Wearable device (103).

5. In claim 4, The first reference distance is determined based on a measurable distance by the first distance sensor, The second reference distance is determined based on a distance measurable by the camera system, and The third reference distance is determined based on a distance measurable by the second distance sensor. Wearable device (103).

6. In claim 5, The measurable distances of each of the first distance sensor, the camera system, and the second distance sensor include a distance where the error between the estimated distance and the actual distance is less than a reference difference. Wearable device (103).

7. In claim 1, The cycle of acquiring the images through the above camera system has a first time interval, The period of obtaining the first distance information through the first distance sensor has a second time interval that is longer than the first time interval, and The period of obtaining the second distance information through the second distance sensor has a third time interval that is longer than the second time interval. Wearable device (103).

8. In claim 7, In a frame where the first time interval overlaps the second time interval, if the distance is less than the first reference distance, the corrected distance is calculated based on the first distance information, and In another frame where the first time interval does not overlap with the second time interval, if the distance is less than the first reference distance, the corrected distance is adjusted from the distance using the first distance information, Wearable device (103).

9. In claim 2, The above instructions, when individually or collectively executed by the at least one processor (510), cause the wearable device (103) to: If the distance is greater than or equal to a fourth reference distance that is less than the first reference distance and less than the first reference distance, cause the corrected distance from the reference position to the external object to be identified based on the first distance information, the second distance information, and the distance, If the distance is greater than or equal to the fourth reference distance and less than or equal to the first reference distance, the corrected distance is identified by applying a first weight to the first distance information and a second weight to the second distance information. If the distance is greater than or equal to the first reference distance and less than or equal to the second reference distance, the corrected distance is identified by applying a third weight to the first distance information and a fourth weight to the second distance information. The first weight exceeds the second weight, and The third weight is less than the fourth weight, Wearable device (103).

10. In claim 9, The fourth reference distance corresponds to the minimum measurable distance to the external object using the images from the plurality of cameras of the camera system. Wearable device (103).

11. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (510), cause the wearable device (103) to: Obtaining a saliency map in which at least a portion of the image is visually emphasized according to the possibility that the gaze of the user of the wearable device (103) is located, Based on the above salience map, identify the weighted region, and Causing to determine a first weight for the first distance information and a second weight for the second distance information for the weighted area, The first weight and the second weight are used to identify the corrected distance. Wearable device (103).

12. In claim 11, The above weighted areas are: identified based on the above projection map and the above distance, Identified based on the above projection map and the gaze region according to the above gaze, Wearable device (103).

13. In claim 1, Each of the above multiple cameras includes an RGB (red-green-blue) camera, The first distance sensor includes an I-ToF (indirect-time of flight) sensor, and The second distance sensor includes a D-ToF (direct-time of flight) sensor. Wearable device (103).

14. In a method performed by a wearable device (103), An operation of identifying a distance from a reference position to an external object within the external environment by using images of the external environment acquired through a camera system including multiple cameras of the wearable device (103); If the distance is greater than or equal to a first reference distance and less than a second reference distance that exceeds the first reference distance, an operation of identifying first distance information of the external object acquired through a first distance sensor of the wearable device (103), second distance information of the external object acquired through a second distance sensor of the wearable device (103), and a corrected distance from the reference position to the external object based on the distance; and Including the action of displaying one or more images that are modified so that a visual object image corresponding to said external object appears to be located at a depth according to said corrected distance. method.

15. A non-transitory computer-readable storage medium, when individually or collectively executed by at least one processor (510) of a wearable device (103) including a plurality of sensors (530) including a first distance sensor and a second distance sensor, a camera system including a plurality of cameras (540), and a display system including a first display and a second display, causes the wearable device (103) to: Using images of the external environment acquired through the above camera system, the distance from a reference position to an external object within the external environment is identified, Compare with criteria related to said distance to said external object and measurable distances of said sensors; Identifying the corrected distance from the reference position to the external object using the above distance and the sensor information of the above sensors, and storing one or more programs storing instructions causing the display system to display one or more images, wherein the images of a visual object corresponding to the external object are modified to appear positioned at a depth according to the corrected distance; A non-transitory computer-readable storage medium.

Images