Electronic device and method for tracking external object in virtual environment

Abstract

This wearable device may comprise: a memory for storing instructions; a display; a communication circuit; and at least one processor. The instructions may, when executed individually or collectively by the at least one processor, cause the wearable device to: identify an external object subject to position tracking on the basis of the communication circuit; display, through the display, a screen including a virtual environment in a virtual reality (VR) mode executed on the basis of identification of an event; identify whether the position of the external object is within a reference identification area including an identification area for loss detection in the VR mode; on the basis of identification that the position is within the reference identification area, display the screen including a visual object for the external object through the display; and on the basis of identification that the position is outside the reference identification area, switch the VR mode to an augmented reality (AR) mode.

Description

Electronic device and method for tracking external objects in a virtual environment

[0001] The following descriptions relate to electronic devices and methods for tracking external objects in a virtual environment.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide extended reality services that display computer-generated information in conjunction with external objects in the real world or virtual objects in the virtual world. The electronic devices may include wearable devices that can be worn by a user. For example, the electronic devices may include user equipment, AR glasses, VR glasses, and / or a head-mounted device (HMD) (e.g., a video see-through (VST) HMD, an optical see-through (OST) HMD).

[0003] A wearable device may include a memory storing instructions and including one or more storage media. The wearable device may include a display. The wearable device may include communication circuitry. The wearable device may include at least one processor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify an external object that is a target of position tracking based on the communication circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a screen including a virtual environment through the display in a virtual reality (VR) mode that is executed based on the identified event. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether the location of the external object is within a reference identification area including an identification area for loss detection in the VR mode. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, through the display, the screen including a visual object for the external object based on identifying that the location is within the reference identification area.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to switch the VR mode to an augmented reality (AR) mode based on identifying that the location is outside the reference identification area.

[0004] According to one embodiment, the external object may include an electronic device that provides information for location tracking in a state in which a connection with the wearable device is established, or an electronic device that provides information for location tracking by transmitting and receiving signals in a state in which a connection with the wearable device is not established.

[0005] A method performed by a wearable device may include an operation of identifying an external object that is a target of position tracking. The method may include an operation of displaying a screen including a virtual environment within a virtual reality (VR) mode that is executed based on the identification of an event. The method may include an operation of identifying whether a location of the external object is within a reference identification area that includes an identification area for loss detection of the VR mode. The method may include an operation of displaying the screen including a visual object for the external object based on identifying that the location is within the reference identification area. The method may include an operation of switching the VR mode to an augmented reality (AR) mode based on identifying that the location is outside the reference identification area.

[0006] A non-transitory computer-readable storage medium may store one or more programs comprising instructions that, when individually or collectively executed by at least one processor of a wearable device including a display and communication circuitry, cause the wearable device to identify an external object that is a target of position tracking based on the communication circuitry. The non-transitory computer-readable storage medium may store one or more programs comprising instructions that, when individually or collectively executed by the at least one processor, cause the wearable device to display, through the display, a screen including a virtual environment in a virtual reality (VR) mode that is executed based on identifying an event. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor, cause the wearable device to identify whether the location of the external object is within a reference identification area including an identification area for loss detection in the VR mode. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor, cause the wearable device to display, through the display, the screen including a visual object for the external object based on identifying that the location is within the reference identification area.The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor, cause the wearable device to switch the VR mode to an augmented reality (AR) mode based on identifying that the location is outside the reference identification area.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2A illustrates an example of a perspective view of a wearable device according to various embodiments.

[0009] FIG. 2b illustrates an example of one or more hardware elements arranged within a wearable device according to various embodiments.

[0010] FIGS. 3A and 3B illustrate examples of the appearance of a wearable device according to various embodiments.

[0011] Figure 4a illustrates examples of areas set with respect to a wearable device.

[0012] FIG. 4b illustrates an example of how a wearable device changes a visual object or operation mode based on lost detection of an external object.

[0013] FIG. 5 illustrates an exemplary block diagram of a wearable device that performs loss detection for external objects.

[0014] Figure 6 illustrates an example of an operational flow for a method for a wearable device to perform loss detection for an external object.

[0015] Figures 7a to 7d illustrate examples of areas set for loss detection of external objects depending on the operation mode.

[0016] Figure 8 illustrates an example of a wearable device displaying a visual object to alert of a risk of loss of an external object.

[0017] Figure 9 illustrates an example of a wearable device changing its operating mode.

[0018] FIG. 10 illustrates an example of an operational flow for how a wearable device changes a visual object or operational mode based on loss detection for an external object.

[0019] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0020] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0021] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled. However, this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." Conditions described as "more than" may be replaced with "more than," conditions described as "less than," and conditions described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of the elements from A (including A) to B (including B).

[0022] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0023] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0024] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0025] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0026] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0027] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0028] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0029] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0030] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0031] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0032] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0033] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0034] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0036] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0037] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0038] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0040] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0041] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0042] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0043] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0044] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0045] FIG. 2A illustrates an example of a perspective view of a wearable device according to various embodiments. FIG. 2B illustrates an example of one or more hardware components arranged within a wearable device according to various embodiments.

[0046] According to one embodiment, the wearable device (103) may have the form of glasses that are wearable on a body part of the user (e.g., the head). The wearable device (103) of FIGS. 2A and 2B may be an example of the electronic device (101) of FIG. 1. The wearable device (103) may include a head-mounted display (HMD). For example, the housing of the wearable device (103) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (103) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0047] Referring to FIG. 2A, according to one embodiment, a wearable device (103) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0048] According to one embodiment, a wearable device (103) can be worn on a part of a user's body. The wearable device (103) can provide extended reality (XR) to a user wearing the wearable device (103). For example, the extended reality can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality. For example, the wearable device (103) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 263) of FIG. 2B.

[0049] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0050] Referring to FIG. 2B, at least one display (250) can provide visual information transmitted from external light to a user through a lens included in at least one display (250), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of at least one display (250). When a user wears the wearable device (103), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0051] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (103) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0052] The wearable device (103) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (103) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (103) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (103) can view an image displayed on at least one display (250).

[0053] According to one embodiment, the frame (200) may be configured as a physical structure that allows the wearable device (103) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that, when the user wears the wearable device (103), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0054] Referring to FIG. 2A, the frame (200) may include a region (220) that is in contact with at least a portion of a user's body when the user wears the wearable device (103). For example, the region (220) of the frame (200) that is in contact with a portion of the user's body may include a region that is in contact with a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (103) makes contact with. According to one embodiment, the frame (200) may include a nose pad (210) that is in contact with a portion of the user's body. When the wearable device (103) is worn by the user, the nose pad (210) may be in contact with a portion of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another portion of the user's body that is distinct from the portion of the user's body.

[0055] For example, the frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (103) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200).

[0056] According to one embodiment, the wearable device (103) may include hardwares (e.g., hardwares to be described later in FIG. 5) that perform various functions. For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (200).

[0057] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (103) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B , but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B . When the number of microphones (265) included in the wearable device (103) is two or more, the wearable device (103) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (200).

[0058] According to one embodiment, at least one optical device (282, 284) can project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) can be a projector. At least one optical device (282, 284) can be disposed adjacent to at least one display (250) or can be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (103) can include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0059] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (103). For example, the wearable device (103) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (103) can perform gaze interaction with at least one object using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (103) can express a portion corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (103) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a first region related to the gaze and a second region distinguished from the first region within the image may be different from each other. For example, if the wearable device (103) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1).An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or may be positioned toward both eyes.

[0060] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image in order to implement augmented reality or mixed reality content. The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including an image of the specific object acquired using the capturing camera (260-4) is superimposed on a virtual image provided through at least one optical device (282, 284). The wearable device (103) can compensate for depth information (e.g., the distance between the wearable device (103) and an external object acquired using a depth sensor) using the image acquired using the capturing camera (260-4). The wearable device (103) can perform object recognition using the image acquired using the capturing camera (260-4). The wearable device (103) may perform a pass-through function to display an image acquired through a photographing camera (260-4) by overlaying it on at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the photographing camera (260-4) may be positioned on a bridge (203) positioned between a first rim (201) and a second rim (202).

[0061] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (103) and thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (103) looks straight ahead, the wearable device (103) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (103) is positioned.

[0062] The motion recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The motion recognition camera (260-2, 260-3) can recognize the user's motion (gesture recognition), obtain a signal corresponding to the motion, and provide a display corresponding to the signal on at least one display (250). The wearable device (103) can identify the signal corresponding to the motion, and perform a designated function based on the identification. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The wearable device (103) can perform gesture recognition and / or object tracking functions using motion recognition cameras (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) can be positioned on the first rim (201) and / or the second rim (202).

[0063] The camera (260) included in the wearable device (103) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (103) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (103) can identify an external object based on a sensor for identifying the distance between the wearable device (103) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (103) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (103).

[0064] Although not shown, in one embodiment, the wearable device (103) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (200) and the hinge units (206, 207).

[0065] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (103). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0066] The antenna module (275) can transmit signals or power to the outside of the wearable device (103), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0067] The speaker (255) can output an audio signal to the outside of the wearable device (103). The audio output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (103). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus positioned adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus positioned adjacent to the user's right ear.

[0068] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state, in order to visually provide information regarding a specific state of the wearable device (103) to the user. For example, when the wearable device (103) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0069] Referring to FIG. 2B, according to one embodiment, a wearable device (103) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (103) (e.g., hardwares to be described later in FIG. 5) may be disposed on the PCB (290). The wearable device (103) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0070] According to one embodiment, the wearable device (103) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (103) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (103). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (103) may identify a user's motion and / or gesture performed to execute or cease a specific function of the wearable device (103) based on the IMU.

[0071] FIGS. 3A and 3B illustrate examples of the appearance of a wearable device according to various embodiments.

[0072] The wearable device (103) of FIGS. 3A and 3B may be an example of the electronic device (101) of FIG. 1. According to one embodiment, an example of the appearance of a first side (310) of a housing of the wearable device (103) may be illustrated in FIG. 3A, and an example of the appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B.

[0073] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (103) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (103) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (103) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0074] According to one embodiment, the wearable device (103) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (103) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (103) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6).

[0075] Referring to FIG. 3b, a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor (330)) for obtaining information related to the external environment of the wearable device (103) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3a. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0076] For example, using cameras (260-11, 260-12), the wearable device (103) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be placed on the second face (320) of the wearable device (103) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be placed on the second face (320) of the wearable device (103) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0077] According to one embodiment, the wearable device (103) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (103) and an external object. Using the depth sensor (330), the wearable device (103) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (103). Although not shown, a microphone may be disposed on the second face (320) of the wearable device (103) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0078] Although not illustrated in FIGS. 2A to 3B , according to one embodiment, the wearable device (103) may include at least one button. For example, the at least one button may be referred to as a physical button or a touch pad. For example, the at least one button may be used to switch the operating mode of the wearable device (103). For example, the mode may include an AR mode and a VR mode.

[0079] Figure 4a illustrates examples of areas set with respect to a wearable device.

[0080] The wearable device (103) of FIG. 4a may represent an example of the electronic device (101) of FIG. 1 or the wearable device (103) of FIGS. 2a to 3b.

[0081] FIG. 4A illustrates an example (400) of a plurality of areas (401, 402, 403) set for a wearable device (103). Referring to example (400), the wearable device (103) can detect the locations of external objects (404-1, 404-2, 404-3). The external objects may include electronic devices or non-electronic devices. For example, the external object (404-1) may include a smartphone. For example, the external object (404-2) may include a bag. For example, the external object (404-3) may include a tablet PC. However, the embodiments of the present disclosure are not limited thereto.

[0082] For example, the external object, which is an electronic device, may also be referred to as a target object, an external object, a location tracking object, or a lost detection object. For example, the wearable device (103) may identify the locations of the external objects (404-1, 404-2, 404-3) by using at least one of a sensor (e.g., a sensor module (176) of FIG. 1), a camera (e.g., a camera module (180) of FIG. 1), or a communication circuit (e.g., a communication module (190) of FIG. 1) included in the wearable device (103). Although not illustrated in FIG. 4A, the wearable device (103) may track the locations of the external objects (404-1, 404-2, 404-3) through an external electronic device connected via the communication circuit. In other words, by receiving the results of location tracking of external objects (404-1, 404-2, 404-3) from the external electronic device, the wearable device (103) can identify (or detect) the locations of the external objects (404-1, 404-2, 404-3). For example, the external electronic device may also be referred to as a source device, a helper device, a server, or a tracking device.

[0083] According to one embodiment, the wearable device (103) may set a plurality of areas (401, 402, 403). For example, the plurality of areas (401, 402, 403) may include areas (402, 403) for identifying a risk of loss according to the locations of external objects (404-1, 404-2, 404-3) and an area (401) in which a user of the wearable device (103) can safely move. For example, the areas (402, 403) for identifying the risk of loss may be determined according to an operation mode of the wearable device (103). For example, the operation mode may include a VR mode and an AR mode. The VR mode may be referred to as a first mode. The AR mode may be referred to as a second mode or a see-through mode. For example, region (402) may represent an identification region according to the VR mode. For example, region (402) may be referred to as a first identification region, a VR identification region, or a VR detection region. For example, region (403) may represent another identification region according to the AR mode. For example, region (403) may be referred to as a second identification region, an AR identification region, an AR detection region, or a basic detection region.

[0084] According to one embodiment, region (401) may be included in each of region (402) and region (403). For example, the size of region (401) may be smaller than the size of region (402) and the size of region (403). Also, for example, region (402) may be included in region (403). For example, the size of region (402) may be smaller than the size of region (403). Referring to example (400), region (401), region (402), and region (403) may be set for the wearable device (103). In example (400), an example in which region (401), region (402), and region (403) are formed in a circle is illustrated, but this is merely for convenience of explanation, and embodiments of the present disclosure are not limited thereto. For example, the areas (401), (402), and (403) may be formed in shapes different from a circle.

[0085] Referring to example (400), the wearable device (103) can identify that an external object (404-1) is located within an area (402). Furthermore, the wearable device (103) can identify that an external object (404-2) is located within an area (403). Furthermore, the wearable device (103) can identify that an external object (404-3) is located outside an area (403). For example, the wearable device (103) can identify that the risk of loss of the external object (404-1) is low while the VR mode is running. Conversely, the wearable device (103) can identify that the risk of loss of the external object (404-2) and the external object (404-3) are high while the VR mode is running. For example, the wearable device (103) may identify that the external object (404-1) and the external object (404-2) have a low risk of loss while the AR mode is running. In contrast, the wearable device (103) may identify that the external object (404-3) has a high risk of loss while the AR mode is running. In example (400), an example of detecting a risk of loss (or detecting loss) based on different positions of three external objects (404-1, 404-2, 404-3) is described, but the embodiments of the present disclosure are not limited thereto. For example, the position of the external object (404-1) may change based on the movement (or movement) of the external object (404-1), and accordingly, the wearable device (103) may identify a risk of loss of the external object (404-1).

[0086] As described above, the wearable device (103) can notify the user of the wearable device (103) of information regarding the identified risk of loss. Specific details related to this are described below in FIG. 4b.

[0087] FIG. 4b illustrates an example of how a wearable device changes a visual object or operation mode based on lost detection of an external object.

[0088] FIG. 4b illustrates examples (405, 406, 407, 408) of a method for notifying the user of the wearable device (103) of the risk of loss by identifying the risk of loss to an external object while the wearable device (103) is executing the VR mode.

[0089] Referring to example (405), the wearable device (103) may display a screen including a virtual environment (410) through a display while the VR mode is being executed. For example, the screen may include at least a portion of the virtual environment (410). The at least portion may represent the virtual environment (410) within the field of view (FoV) of the wearable device (103). For example, the wearable device (103) may execute the VR mode based on identifying an event for executing the VR mode. For example, the event may include at least one of an input to a physical button included in the wearable device (103) for executing the VR mode, an input to a portion of the display, or a gesture by a user of the wearable device (103) on the FoV. For example, the physical button may include a button used for switching between an AR mode and a VR mode.

[0090] According to one embodiment, the wearable device (103) may provide an AR mode based on identifying that the user is wearing the wearable device (103). For example, the wearable device (103) may have the AR mode as its default mode. The wearable device (103) may execute the VR mode based on identifying the event. However, the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may have the VR mode as its default mode. Alternatively, for example, the wearable device (103) may have a transparent mode that is different from the VR mode and the AR mode as its default mode. For example, the transparent mode may indicate a state in which no virtual information about the external environment is provided, such as when wearing general glasses.

[0091] Although not shown in Example (405), the wearable device (103) may perform position tracking for the external object while displaying the screen including the virtual environment (410) through the VR mode. For example, the wearable device (103) may directly identify the position of the external object based on the communication circuit included in the wearable device (103). Alternatively, for example, the wearable device (103) may identify the position of the external object using an external electronic device connected to the wearable device (103) through the communication circuit. In Example (405), by identifying the position within the area (402) of FIG. 4A, the wearable device (103) may only display the screen including the virtual environment (410).

[0092] According to one embodiment, the wearable device (103) may identify the location based on a positioning technique while the AR mode is running. For example, the positioning technique may include Wi-Fi-based positioning, BLE-based positioning, IoT (internet of things) server-based positioning, UWB (ultra-wideband)-based positioning, ToF (time of flight) camera-based positioning, and GPS-based positioning. Alternatively, the wearable device (103) may identify the location while the VR mode is running based further on at least one of information acquired from an AP (access point) to which the external object and the wearable device (103) are connected, or information about a signal received from the external object, together with the positioning technique. Specific details related thereto are described below in FIG. 6. As described above, the wearable device (103) may use another location identification method together with the location positioning technique to identify the location more accurately in the VR mode, unlike the AR mode that provides the real world through a display.

[0093] Referring to examples (406) and (407), the wearable device (103) may display a visual object (420) or a visual object (430) for the external object based on identifying that the location of the external object is outside of the area (402) of FIG. 4A. For example, the wearable device (103) may display the screen including the visual object (420) or the visual object (430) for the external object based on identifying that the location is outside of the area (402) and within the reference identification area. For example, the reference identification area may represent an area used to identify a risk of loss within the VR mode. For example, the reference identification area may include the area (402) of FIG. 4A and may be included in the area (403) of FIG. 4A. In other words, the reference identification area may define a range between the areas (402) and (403).

[0094] According to one embodiment, the wearable device (103) can track the external object. For example, the wearable device (103) can track the external object using an image acquired through a camera included in the wearable device (103). The wearable device (103) can map between the external object identified through the communication circuit and the external object tracked through the image. In other words, the wearable device (103) can identify the external object in the image based on the location identified through the communication circuit. The wearable device (103) can display a visual object (420) or a visual object (430) in a display area of ​​the display of the wearable device (103) corresponding to the location of the external object in the image.

[0095] According to one embodiment, the visual object (420) or the visual object (430) may at least partially overlap with the virtual environment (410). For example, the visual object (420) or the visual object (430) may be displayed in a floating state on the virtual environment (410). For example, the visual object (420) may include text for notifying the risk of loss. For example, the visual object (430) may include a rendered image representing the external object. In addition, the visual object (430) may further include text for notifying the image and movement of the external object. However, the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may also display a rendered image or the real environment for an area of ​​the real environment that includes the external object. For example, the rendered image or the real environment may at least partially overlap with the virtual environment (410).

[0096] In examples (406) and (407), visual objects (420, 430) displayed by the wearable device (103) are depicted when the location is located outside the area (402) and within the reference identification area. The display of visual objects (420, 430) may indicate that there is a possibility of loss of the external object depending on the location. Accordingly, the user can recognize the possibility of loss of the external object.

[0097] Alternatively, referring to example (408), the wearable device (103) may execute the AR mode, which is changed from the VR mode, based on identifying that the location of the external object is outside the reference identification area. For example, the wearable device (103) may terminate the VR mode and execute the AR mode based on identifying that the location is outside the reference identification area. For example, the wearable device (103) may provide a real environment (440) by executing the AR mode. For example, the real environment (440) may be provided through the display of the wearable device (103). For example, the real environment (440) may be provided by being transmitted through the display.

[0098] In example (408), a real environment (440) provided by the wearable device (103) is depicted when the location is outside the reference identification area. The provision of the real environment (440) may indicate a high possibility of loss or loss of the external object depending on the location. Accordingly, the user may recognize that there is a higher risk of loss than the possibility of loss of the external object in examples (406) and (407).

[0099] Although not illustrated in FIG. 4B, the wearable device (103) may be connected to an external electronic device to identify the location of the external object. For example, the wearable device (103) may be connected to the external electronic device via short-range communication. For example, the short-range communication may include BT (Bluetooth), BLE (Bluetooth low energy), Wi-Fi (Wireless fidelity), or UWB (ultra-wideband). However, the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may be connected to the external electronic device based on a wireless communication or wired communication technology that is different from the short-range communication.

[0100] A method for detecting loss of the external object may utilize an electronic device and a tracker. For example, if the external object is an electronic device, the external object may identify itself as lost and notify a server connected to the external object of the loss. Thereafter, a user of the external object may access the server and locate the external object based on the loss information. However, since the external object must directly notify the server of the information, it is limited to only certain devices. Due to this limitation, a separate electronic device, a tracker, may be utilized for loss detection (or location tracking). The tracker may advertise a packet containing identification information (e.g., an ID) of the tracker when communicating with an electronic device (e.g., a smartphone) and the communication connection with the electronic device is terminated. Another electronic device receiving the packet may provide the identification information and location information to the server. A user of the electronic device can identify the location of an object that includes or is attached to the tracker by accessing the server. However, even when using the tracker, the tracker's ability to notify of a loss status may be limited due to the tracker's limited battery capacity. Furthermore, the tracker cannot identify whether the tracker is lost while it is not connected to the electronic device. The tracker's sending of an advertising packet to the electronic device after the tracker has been lost may be a post-loss measure after the tracker has been lost.

[0101] In addition, when the wearable device (103) providing the above-mentioned extended reality provides a VR environment, there is a problem in that it is difficult for the user of the wearable device (103) to accurately recognize the user's external situation (or the real world).

[0102] An electronic device and method according to embodiments of the present disclosure provide a technology for controlling a loss detection (or location tracking) operation for an external object according to the operating mode of the wearable device (103) capable of providing extended reality. For example, the external object may include an object representing an electronic device or a non-electronic device. The electronic device and method according to embodiments of the present disclosure can identify the location of a designated external object and identify a risk of loss even while the user wearing the wearable device (103) is using VR mode. The electronic device and method according to embodiments of the present disclosure can change a visual object or an operating mode according to the identified degree of risk of loss. Accordingly, the electronic device and method according to embodiments of the present disclosure can provide information on the possibility of loss or whether the object has been lost in a situation where the user has difficulty checking the external environment. In addition, the electronic device and method according to embodiments of the present disclosure can improve the usability of the wearable device (103).

[0103] FIG. 5 illustrates an exemplary block diagram of a wearable device that performs loss detection for external objects.

[0104] FIG. 5 illustrates an exemplary block diagram of a wearable device (103). Referring to FIG. 5, the wearable device (103) may be connected to an external object (520) that is a target of location tracking. In this case, the external object (520) may be an electronic device. However, the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may identify an external object (520) designated by a user of the wearable device (103), even if the external object (520) is a non-electronic object. For example, the wearable device (103) may identify the location of the external object (520) based on a sensor (505), a camera (507), or a communication circuit (509) for the external object (520). In addition, the wearable device (103) may be connected to an external electronic device (530). The wearable device (103) may also identify the location of an external object (520) using a connected external electronic device (530). FIG. 5 illustrates an example of a wearable device (103) connected to one external object (520) and one external electronic device (530), but the embodiments of the present disclosure are not limited thereto. For example, the number of external electronic devices (530) connected to the wearable device (103) may be zero or two or more. Furthermore, for example, the number of external objects (520) identified or connected to the wearable device (103) may be two or more. The wearable device (103) of FIG. 5 may represent an example of the electronic device (101) of FIG. 1 or the wearable device (103) of FIGS. 2A to 3B.

[0105] Referring to FIG. 5, an exemplary situation is illustrated in which a wearable device (103), an external object (520), and an external electronic device (530) are connected to each other based on a wired network and / or a wireless network. For example, the wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), or a combination thereof. For example, the wireless network may include a network such as long term evolution (LTE), 5g new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. Although the wearable device (103), the external object (520), and the external electronic device (530) are illustrated as being directly connected, the wearable device (103), the external object (520), and the external electronic device (530) may be indirectly connected via one or more routers and / or access points (APs). In other words, the wearable device (103) is exemplified as being directly connected to the communication circuit (not shown) of the external object (or the communication circuit (not shown) of the external electronic device (530)) through the communication circuit (509), but the embodiments of the present disclosure are not limited thereto.

[0106] Referring to FIG. 5, according to one embodiment, a wearable device (103) may include at least one of a processor (501), a display (503), a sensor (505), a camera (507), a communication circuit (509), and a memory (511). The processor (501), the display (503), the sensor (505), the camera (507), the communication circuit (509), and the memory (511) may be electronically and / or operably coupled with each other by a communication bus. Hereinafter, operably coupled hardware components may mean that a direct connection or an indirect connection is established between the hardware components, either wired or wireless, such that a second hardware component is controlled by a first hardware component among the hardware components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware components illustrated in FIG. 5 (at least a portion of the processor (501), the communication circuit (509), and the memory (511)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware components included in the wearable device (103) is not limited to those illustrated in FIG. 5. For example, the wearable device (103) may include only some of the hardware components illustrated in FIG. 5.

[0107] According to one embodiment, the processor (501) of the wearable device (103) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), and / or a neural processing unit (NPU). The number of processors (501) may be one or more. For example, the processor (501) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core. The processor (501) of FIG. 5 may include at least a part of the processor (120) of FIG. 1.

[0108] For example, the processor (501) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform the various functions.

[0109] According to one embodiment, a display (503) of a wearable device (103) can output visualized information (e.g., a visual object of FIG. 8) to a user. The number of displays (503) included in the wearable device (103) may be one or more. For example, the display (503) may be controlled by a processor (501) and / or a graphic processing unit (GPU) (not shown) to output visualized information to a user. The display (503) may include a flat panel display (FPD) and / or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), a digital mirror device (DMD), one or more light emitting diodes (LEDs), and / or micro LEDs. The LEDs may include organic LEDs (OLEDs). The display (503) of FIG. 5 may include at least a part of the display module (160) of FIG. 1.

[0110] In one embodiment, light transmission may occur through at least a portion of the display (503). The wearable device (103) may provide a user experience related to augmented reality by providing a combination of light output through the display (503) and light transmitted through the display (503) to the user. As described above with reference to FIGS. 2A and 2B and / or 3A and 3B , the display (503) of the wearable device (103) according to one embodiment may have a structure for covering the entire FoV of the user, or radiating light toward the FoV of the user, while being worn on a body part of the user, such as the head. Although not illustrated, the wearable device (103) may include other output means for outputting information in forms other than visual or auditory forms. For example, the wearable device (103) may include at least one speaker for outputting an audio signal, and / or a motor (or actuator) for providing haptic feedback based on vibration. In the above example, a display (503) for providing a user experience related to the augmented reality is exemplified, but the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may render an image for providing a user experience related to virtual reality, and display the rendered image through the display (503).

[0111] Although not illustrated in FIG. 5, the wearable device (103) according to one embodiment may include a sensor (505) and a camera (507). For example, the wearable device (103) may include a sensor (505) for identifying a designated event, another designated event, and a user's input. For example, the sensor (505) may include an IMU (or IMU sensor), a biometric sensor, an acoustic sensor (or a microphone (or mic)), a gyro sensor, a gravity sensor, and / or an acceleration sensor. For example, the wearable device (103) may include a camera (507) (or an image sensor) for recognizing a real environment. The sensor (505) of FIG. 5 may include at least a portion of the sensor module (176) of FIG. 1. The camera (507) of FIG. 5 may include at least a portion of the camera module (180) of FIG. 1. For example, the wearable device (103) may identify the location of an external object (520) through the sensor (505) and / or the camera (507). In this case, the external object (520) may represent an object other than an electronic device (i.e., an object that is a non-electronic device). The external object (520) may be a target designated by the user of the wearable device (103). Alternatively, for example, the wearable device (103) may recognize an external object (520) in an image acquired through the camera (507) based on the location of the external object (520) acquired through the communication circuit (509). For example, the wearable device (103) may map the location and the external object (520) in the image.

[0112] According to one embodiment, the communication circuit (509) of the wearable device (103) may include hardware for supporting transmission and / or reception of electrical signals between the wearable device (103) and an external object (520) (or an external electronic device (530)). The communication circuit (509) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (509) may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth (BT), Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G new radio (NR), and wireless-fidelity (Wi-Fi). The communication circuit (509) of FIG. 5 may include at least a part of the communication module (190) and / or the antenna module (197) of FIG. 1.

[0113] According to one embodiment, the memory (511) of the wearable device (103) may include a hardware component for storing data and / or instructions input to and / or output from the processor (501). The memory (511) may include, for example, a volatile memory such as a random-access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multimedia card (eMMC). The memory (511) of FIG. 5 may include at least a part of the memory (130) of FIG. 1.

[0114] Additionally, although not illustrated in FIG. 5, the wearable device (103) according to one embodiment may include an output means for outputting information in a form other than a visualized form. For example, the wearable device (103) may include a speaker for outputting an acoustic signal. For example, the wearable device (103) may include a motor for providing haptic feedback based on vibration.

[0115] Referring to FIG. 5, according to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (501) of the wearable device (103) may be stored in the memory (511) of the wearable device (103). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, when an application is installed in an electronic device (e.g., a wearable device (103)), it may mean that one or more instructions provided in the form of an application are stored in the memory (511), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension designated by the operating system of the wearable device (103)). According to one embodiment, the wearable device (103) may perform the operations of FIGS. 6 and 10 by executing one or more instructions stored in the memory (511).

[0116] According to one embodiment, an external object (520) may be a target of position tracking. For example, the external object (520) may be an electronic device or a non-electronic device.

[0117] For example, an external object (520), which is an electronic device, can establish a connection with a wearable device (103). The wearable device (103) can identify the location of the external object (520) based on the connection. For example, the wearable device (103) can obtain location information through the connection with the external object (520) within a specified distance, and can identify the location of the external object (520) based on the location information. However, the embodiments of the present disclosure are not limited thereto. For example, the external object (520) with which the connection has been established can be understood as being located within the specified distance of the wearable device (103). Alternatively, the external object (520), which is an electronic device, can transmit and receive signals with the wearable device (103) without establishing a connection with the wearable device (103). For example, the wearable device (103) can transmit a request signal to the external object and receive a response signal from the external object (520). Based on the request signal and the response signal, the wearable device (103) can identify the location of the external object (520).

[0118] For example, the wearable device (103) can identify the location of the external object by receiving an advertising signal transmitted by the external object in the state where the connection is established. Location tracking performed in the state where the connection is established may be referred to as a device finding service based on a connection method. Alternatively, for example, the wearable device (103) can also identify the location of the external object by receiving the advertising signal transmitted by the external object in the state where the connection is not established. Location tracking performed in the state where the connection is not established may be referred to as a device finding service based on a non-connection method. Although not described in the above example, the location tracking may further include location tracking that establishes and performs a connection only when necessary. In addition, for example, in the device finding service, an external electronic device (e.g., external electronic device (530)) may provide information to a server notifying that the external object is lost based on a signal received from the external object. The wearable device (103) can identify the location of the external object based on the information provided to the server.

[0119] For example, an external object (520), which is a non-electronic device, can be identified by a wearable device (103). For example, the wearable device (103) can identify an external object (520) designated by a user based on at least one of a sensor (505), a camera (507), or a communication circuit (509). For example, the wearable device (103) can identify the location of the external object (520) based on information acquired using at least one of a sensor (505), a camera (507), or a communication circuit (509).

[0120] According to one embodiment, the external electronic device (530) may be a subject that performs location tracking. For example, the external electronic device (530) may perform location tracking for an external object (520). For example, the external electronic device (530) may be connected to a wearable device (103). The wearable device (103) may perform location tracking for the external object (520) using the external electronic device (530). For example, the external electronic device (530) may provide the result of the location tracking to the wearable device (103). The external electronic device (530) may provide the result of the location tracking. The external electronic device (530) may be referred to as a source device, a helper device, a server, or a tracking device. For example, the external electronic device (530) may include at least one of a user equipment, an electronic device (or wearable device) having the same account information as the wearable device (103), or a server that manages another external electronic device.

[0121] Figure 6 illustrates an example of an operational flow for a method for a wearable device to perform loss detection for an external object.

[0122] At least some of the methods of FIG. 6 may be performed by the wearable device (103) of FIG. 5. For example, at least some of the methods may be controlled by the processor (501) of the wearable device (103). In the embodiments described below, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0123] Referring to FIG. 6, in operation (600), the wearable device (103) can identify an external object. The external object may represent a target of location tracking. For example, the wearable device (103) can identify the external object based on the communication circuit (509). At this time, the external object may be an electronic device. For example, the wearable device (103) can identify the external object that is connectable or connected based on the communication circuit (509). Alternatively, for example, the wearable device (103) can identify the external object designated by a user of the wearable device (103). For example, the user can designate the external object based on at least a portion of an input to the wearable device (103). For example, the external object may be specified by the user when the wearable device (103) executes VR mode in operation (605).

[0124] For example, the external object, which is an electronic device, can establish a connection with a wearable device (103). The wearable device (103) can identify the location of the external object based on the connection. For example, the wearable device (103) can obtain location information through the connection with the external object within a specified distance, and can identify the location of the external object based on the location information. Alternatively, the external object, which is an electronic device, can transmit and receive signals with the wearable device (103) without establishing a connection with the wearable device (103). For example, the wearable device (103) can transmit a request signal to the external object and receive a response signal from the external object. Based on the request signal and the response signal, the wearable device (103) can identify the location of the external object.

[0125] For example, the wearable device (103) can identify the location of the external object by receiving an advertising signal transmitted by the external object in the state where the connection is established. Location tracking performed in the state where the connection is established may be referred to as a device finding service based on a connection method. Alternatively, for example, the wearable device (103) can also identify the location of the external object by receiving the advertising signal transmitted by the external object in the state where the connection is not established. Location tracking performed in the state where the connection is not established may be referred to as a device finding service based on a non-connection method. Although not described in the above example, the location tracking may further include location tracking that establishes and performs a connection only when necessary. In addition, for example, in the device finding service, an external electronic device (e.g., external electronic device (530)) may provide information to a server notifying that the external object is lost based on a signal received from the external object. The wearable device (103) can identify the location of the external object based on the information provided to the server.

[0126] For example, the external object, which is a non-electronic device, can be identified by the wearable device (103). For example, the wearable device (103) can identify the external object designated by the user based on at least one of the sensor (505), the camera (507), or the communication circuit (509). For example, the wearable device (103) can identify the location of the external object based on information obtained using at least one of the sensor (505), the camera (507), or the communication circuit (509).

[0127] In operation (605), the wearable device (103) may execute a VR mode. For example, the wearable device (103) may execute the VR mode based on identifying an event for executing the VR mode. For example, the event may include at least one of an input to a physical button included in the wearable device (103) for executing the VR mode, an input to a portion of the display, or a gesture by a user of the wearable device (103) on the FoV. For example, the event may be referred to as an event for executing the VR mode, a mode conversion event, a mode change event, or a mode change input. For example, the physical button may include a button used for switching between an AR mode and a VR mode.

[0128] In one embodiment, the wearable device (103) may provide an AR mode based on identifying that the user is wearing the wearable device (103). For example, the wearable device (103) may have the AR mode as its default mode. The wearable device (103) may execute the VR mode based on identifying the event. However, the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may have the VR mode as its default mode.

[0129] In operation (610), the wearable device (103) may identify a first identification area. For example, the wearable device (103) may identify the first identification area based on the execution of the VR mode. For example, the first identification area (e.g., the first identification area (402) of FIG. 4A) may be referred to as an identification area according to the VR mode, an identification area, a VR identification area, or a VR detection area.

[0130] According to one embodiment, the wearable device (103) may identify the first identification area changed from the second identification area of ​​the AR mode based on the execution of the VR mode changed from the AR mode. For example, the second identification area (the second identification area (403) of FIG. 4A) may be referred to as an identification area according to the AR mode, another identification area, an AR identification area, an AR detection area, or a basic detection area. For example, the second identification area may include the first identification area. In other words, the size of the second identification area may be wider than the size of the first identification area.

[0131] According to one embodiment, the first identification area may be identified based on at least one of the complexity of the surrounding area of ​​the wearable device (103), the risk level of the surrounding area, the time at which the VR mode is executed, or the direction of the field of view (FoV) of the wearable device (103). For example, the surrounding area may represent an area within a specified distance with respect to the omnidirectional direction of the wearable device (103). For example, the complexity may be identified based on the presence or absence of an object or obstacle within the surrounding area. For example, the risk level may represent the degree of risk of the object or the obstacle within the surrounding area. For example, the time may include whether the time at which the VR mode is executed is evening or daytime. This is because, if the time is evening, the detection capability of the surrounding area of ​​the wearable device (103) may be reduced, and it may be difficult for the user to identify the external object. Additionally, for example, the FoV direction may indicate the direction in which the FoV of the wearable device (103) faces. For example, a portion of the first identification area corresponding to the FoV direction may be formed wider than another portion corresponding to a direction different from the FoV direction (e.g., an opposite direction). Specific details related thereto are described below in FIG. 7d.

[0132] In operation (615), the wearable device (103) can identify the location of the external object. For example, the wearable device (103) can identify whether the external object is moving.

[0133] According to one embodiment, the wearable device (103) can identify a distance between the wearable device (103) and an external object. For example, the wearable device (103) can identify a location of the external object based on the distance and the direction of the external object.

[0134] For example, the external object may include an electronic device connected to the same node (e.g., an access point (AP)) as the wearable device (103). For example, a plurality of APs may be deployed within a certain delay to provide a location-based service. The deployment of the plurality of APs may be configured according to the range of a service area or the coverage of the AP. In addition, a server managing APs (e.g., a Wi-Fi Positioning System (WPS)) may manage information on the plurality of APs and location information on the plurality of APs. For example, the wearable device (103) may identify the location of the external object connected to the same AP as the wearable device (103) based on information on the plurality of APs and location information on the plurality of APs obtained from the server.

[0135] For example, the external object can be identified based on the strength of the signal received by the wearable device (103). For example, the wearable device (103) can estimate the distance based on the strength and pattern of the received signal for indoor positioning. For example, the strength of the signal can include a received signal strength indicator (RSSI). For example, the RSSI can be an indicator representing the strength of the signal (unit: dBm), where the strength of the signal decreases with distance. The RSSI can be defined as in the following mathematical equation.

[0136]

[0137] The above n may represent an attenuation coefficient, the above A may represent an RSSI value measured at a specific location (e.g., 1 m (meter)) as a reference value, and the above d may represent a distance from the location where the signal is transmitted. For example, the above attenuation constant may change depending on the environment in which the signal is transmitted and received. For example, the wearable device (103) may estimate a distance value based on the RSSI. The wearable device (103) may identify the location of the external object based on the estimated distance value.

[0138] For example, the external object can be identified based on a location positioning technique (e.g., Wi-Fi-based location positioning, BLE-based location positioning, IoT (internet of things) server-based location positioning). For example, the wearable device (103) can identify distance information and direction information for the external electronic device through reception and transmission of high-frequency signals defined in 802.11.ad among Wi-Fi communication standards. Based on the distance information and the direction information, the AP connected to the external electronic device can identify the exact location of the electronic device. For example, the wearable device (103) can identify the location of the external object based on the distance information and the direction information acquired from the AP. In addition, for example, the wearable device (103) can identify the location of the external electronic device based on the angle of arrival (AoA) and angle of departure (AoD) identified using BLE. For example, the wearable device can identify the location of the external object based on the AoA and AoD. In addition, for example, the wearable device (103) can identify the external object based on information obtained from a server that manages IoT devices. For example, the server can pre-designate or identify the locations of external electronic devices, which are IoT devices connected to the server. For example, the wearable device (103) can identify the location of the external object based on the server.

[0139] In addition, for example, the external object may cause the wearable device (103) to identify the external object by providing a signal including information identifying the wearable device (103) to a server (e.g., an IoT server) connected to the external object. For example, the external object may identify a surrounding object based on a camera or a sensor. For example, a UWB sensor (e.g., UWB radar or UWB ranging) may be used to identify the object. In this case, the object may include a wearable device (103) identified based on the camera or the sensor. The external object may provide information about the identified wearable device (103) to the server. For example, the external object may display a user interface through a display. For example, the user interface may be a user interface for the user to confirm whether the device (e.g., wearable device (103)) to be identified through the server (or P2P (peer-to-peer)) is correct. Based on the user's input (or gesture) to the user interface, the external object may provide information to the server. The server may perform a connection (or mapping) between the wearable device (103) and the external object based on the information. At this time, the connection may be accompanied by authentication (e.g., a predetermined pattern) to enhance security. Accordingly, the wearable device (103) may also identify the external object. In the above example, the case where the object is the wearable device (103) is exemplified, but the embodiments of the present disclosure are not limited thereto.For example, by identifying a user of a designated gesture or wearable device (103), the external object can identify the presence of the wearable device (103) and thereby provide the information about the wearable device (103) to the server. The wearable device (103) can also identify the location of the external object based on the information provided from the server.

[0140] According to one embodiment, the wearable device (103) may use at least one of the above methods for identifying the location of the external object. Furthermore, embodiments of the present disclosure are not limited thereto, and other methods for identifying the location of the external object may be used. For example, the wearable device (103) may identify the location based on a location positioning technique while the AR mode is running. For example, the location positioning technique may include Wi-Fi-based location positioning, BLE-based location positioning, IoT (internet of things) server-based location positioning, UWB (ultra-wideband)-based location positioning, ToF (time of flight) camera-based location positioning, and GPS-based location positioning. Alternatively, the wearable device (103) may identify the location while the VR mode is running based on at least one of information obtained from an access point (AP) to which the external object and the wearable device (103) are connected, together with the location positioning technique, or information about a signal received from the external object. For example, the information about the signal may include a difference in the timing at which signals are received, or the reception strength of the signals. Alternatively, for example, the wearable device (103) may identify the location of the external object based on BLE-based location positioning while the AR mode is running, and may identify the location of the external object based on a specific sensor (e.g., a ToF sensor) while the VR mode is running. In other words, a method for identifying the location that is more precise than that in the AR mode may be used in the VR mode.

[0141] According to one embodiment, the wearable device (103) can track the external object. For example, the wearable device (103) can track the external object using an image acquired through a camera (507) included in the wearable device (103). The wearable device (103) can map between the external object identified through the communication circuit (509) and the external object tracked through the image. In other words, the wearable device (103) can identify the external object in the image based on the location identified through the communication circuit (509). The wearable device (103) can display a visual object (hereinafter, the visual object of operation (630)) in a display area within the display (503) of the wearable device (103) corresponding to the location of the external object in the image.

[0142] In operation (620), the wearable device (103) can identify whether the location of the external object is within the first identification area. Based on identifying that the location is within the first identification area in operation (620), the wearable device (103) can perform operation (640). Alternatively, based on identifying that the location is outside the first identification area in operation (620), the wearable device (103) can perform operation (625).

[0143] In operation (625), the wearable device (103) can identify whether the location of the external object is within a reference identification area. According to one embodiment, the reference identification area may, for example, represent an area used to identify a risk of loss within the VR mode. For example, the reference identification area may include the first identification area and be included in the second identification area. In other words, the reference identification area may define a range between the first identification area and the second identification area. For example, the reference identification area may be referred to as a first reference area.

[0144] In operation (625), the wearable device (103) may perform operation (630) based on identifying that the location is within the reference identification area. In operation (625), the wearable device (103) may perform operation (635) based on identifying that the location is outside the reference identification area.

[0145] In operation (630), the wearable device (103) may display a visual object for the external object. For example, the wearable device (103) may display the screen including the visual object for the external object based on identifying that the location is outside the first identification area and within the reference identification area.

[0146] According to one embodiment, the visual object may at least partially overlap with the virtual environment provided within the VR mode. For example, the visual object may be displayed in a floating state on the virtual environment. For example, the visual object may include text to indicate the risk of loss. For example, the visual object may include a rendered image representing the external object. In addition, the visual object may further include text to indicate movement of the image and the external object. However, the embodiments of the present disclosure are not limited thereto. Specific details related thereto are described below in FIG. 8.

[0147] Additionally, according to one embodiment, the wearable device (103) may provide the user with a notification different from the visual object in operation (630). For example, the wearable device (103) may notify the user of a risk of loss by outputting auditory information (e.g., sound). Alternatively, for example, the wearable device (103) may notify the user of a risk of loss by outputting tactile information (e.g., vibration). Alternatively, for example, the wearable device (103) may provide at least one of the auditory information and / or the tactile information together with the visual object.

[0148] In operation (635), the wearable device (103) may switch to the AR mode. For example, the wearable device (103) may switch from the VR mode to the AR mode based on identifying that the location is outside the reference identification area. For example, the wearable device (103) may terminate the VR mode and execute the AR mode based on identifying that the location is outside the reference identification area. For example, the wearable device (103) may provide a real environment by executing the AR mode. For example, the real environment may be provided through the display of the wearable device (103). For example, the real environment may be provided by being transmitted through the display. Specific details related thereto are described below in FIG. 9.

[0149] In operation (640), the wearable device (103) can identify whether the location of the external object is within a movement detection area. For example, the movement detection area can be defined with respect to the location of the external object. According to one embodiment, the movement detection area can be used to identify the loss detection of the external object. For example, the movement detection area can be included in the first identification area. For example, the movement detection area can be referred to as a detection area, an object detection area, or an object movement detection area.

[0150] In operation (640), the wearable device (103) may perform operation (615) based on identifying that the location of the external object is within the movement detection area. According to one embodiment, the wearable device (103) may identify a new location of the external object in operation (615) if it identifies that the location changed as the external object moves is within the movement detection area. Alternatively, in operation (640), the wearable device (103) may perform operation (645) based on identifying that the location of the external object is outside the movement detection area. According to one embodiment, the wearable device (103) may identify that there is a possibility of loss if the location goes out of the movement detection area as the external object moves.

[0151] In operation (645), the wearable device (103) can identify whether the location of the external object is within a reference detection area. According to one embodiment, the reference detection area can be defined with respect to the movement detection area. For example, the reference detection area can include the movement detection area. In addition, the reference detection area can be included in the first identification area. For example, the reference detection area can be referred to as a second reference area.

[0152] In operation (645), the wearable device (103) may perform operation (630) based on identifying the location outside the movement detection area but within the reference detection area. Accordingly, the wearable device (103) may display the visual object for the external object within operation (630). Alternatively, the wearable device (103) may change to the AR mode within operation (635) based on identifying the location outside the reference detection area.

[0153] In FIG. 6, an example of a method for performing operations (640) and (645) is illustrated, but embodiments of the present disclosure are not limited thereto. For example, the method may omit operations (640) and (645). If operations (640) and (645) are omitted, the wearable device (103) may perform operation (615) based on identifying that the location of the external object is within the first identification area in operation (620). According to one embodiment, the wearable device (103) may identify a new location of the external object within operation (615) if it identifies that the location changed as the external object moves is within the movement detection area.

[0154] Referring to the above, an example of a wearable device (103) tracking the location of an external object and performing loss detection is described, but the embodiments of the present disclosure are not limited thereto. According to one embodiment, the wearable device (103) may be connected to an external electronic device and may track the location of the external object through the external electronic device. For example, the wearable device (103) may be connected to the external electronic device through a communication circuit (509). For example, the wearable device (103) may be connected to the external electronic device through short-range communication. For example, the short-range communication may include BT (Bluetooth), BLE (Bluetooth low energy), Wi-Fi (Wireless fidelity), or UWB (ultra-wideband). However, the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may be connected to the external electronic device based on a wireless communication or wired communication technology that is different from the short-range communication. For example, the wearable device (103) may identify the external object connected to the external electronic device. For example, the external object may include an electronic device whose location can be tracked by the external electronic device.

[0155] For example, the location of the external object can be tracked according to a device finding service based on a connection method performed by the external electronic device or a device finding service based on a non-connection method. The wearable device (103) can request location tracking of the external object by providing information about the external object to the external electronic device. In addition, the wearable device (103) can provide information about the first identification area and the reference identification area to the external electronic device based on the execution of the VR mode. Accordingly, the external electronic device can change the settings for the identification area for loss detection of the external object. For example, the external electronic device can set the first identification area and the reference identification area. The external electronic device can identify a location that changes according to the movement of the external object. The external electronic device can identify that the location is located outside the first identification area. For example, the external electronic device may identify that the location is outside the first identification area and within the reference identification area and transmit a first signal to the wearable device (103) accordingly. Alternatively, for example, the external electronic device may identify that the location is outside the reference identification area and transmit a second signal to the wearable device (103) accordingly. The first signal may indicate that the location is outside the first identification area and within the reference identification area. The second signal may indicate that the location is outside the reference identification area. The wearable device (103) that receives the signal as described above may perform operation (630) or operation (635). In addition, the external electronic device may identify the location of the external object and the relative location with respect to the movement detection area and the reference detection area, even if the location that changes according to the movement is within the first identification area.Additionally, the external electronic device may transmit a signal indicating the relative position to the wearable device (103). Accordingly, the wearable device (103) may perform operation (630) or operation (635). Specific details related thereto may be substantially identically applied to the operations (620) to (645) described above.

[0156] Additionally, although not described in FIG. 6, the wearable device (103) may set a function to identify the first identification area and change to the visual object or the AR mode when the VR mode is executed. For example, the wearable device (103) may set the first identification area in response to the execution of the VR mode based on the activation of the function. For example, the function may be activated or deactivated based on the user's input within the settings for a software application providing augmented reality or within the global settings of the wearable device (103). Alternatively, for example, the function may be deactivated based on identifying that another user (e.g., a family member) associated with the user is located in the vicinity of the user. This may be because the possibility of losing the external object is relatively low when the other user is present. In addition, when the function is deactivated, the power consumption of the wearable device (103) may be reduced.

[0157] Figures 7a to 7d illustrate examples of areas set for loss detection of external objects depending on the operation mode.

[0158] Referring to FIG. 7A, an example (701) is illustrated in which a first identification area (402), a reference identification area (710), and a second identification area (403) are set for a wearable device (103). Referring to example (701), the second identification area (403) may include the first identification area (402) and the reference identification area (710). In addition, the reference identification area (710) may include the first identification area (402). In other words, the sizes of the areas may be increased in the order of the first identification area (402), the reference identification area (710), and the second identification area (403). In example (701), an example in which the first identification area (402), the reference identification area (710), and the second identification area (403) are formed in a circle is illustrated, but this is merely for convenience of explanation, and the embodiments of the present disclosure are not limited thereto. For example, the first identification area (402), the reference identification area (710), and the second identification area (403) may be formed in a shape different from a circle.

[0159] Referring to example (701), the wearable device (103) can identify an external object (520) while displaying a virtual environment within a VR mode. For example, the wearable device (103) can identify that the external object (520) is located within the first identification area (402). Based on identifying the location (711) of the external object (520), the wearable device (103) can maintain display of a screen including the virtual environment within the VR mode. In one embodiment, based on identifying that the location (711) of the external object (520) has changed to a location (712) due to movement of the external object (520), the wearable device (103) can display a visual object for the external object (520). For example, the wearable device (103) may display the screen including the visual object for the external object based on identifying a location (712) outside the first identification area (402) and within the reference identification area (710). For example, the visual object may include text for notifying the risk of loss. For example, the visual object may include a rendered image of an area of ​​a real environment including the external object or the real environment. In addition, the visual object may further include text for notifying the movement of the image and the external object. However, the embodiments of the present disclosure are not limited thereto. According to one embodiment, the wearable device (103) may execute an AR mode that changes from the VR mode based on identifying a change from a location (712) to a location (713) due to movement of the external object (520). For example, the wearable device (103) may terminate the VR mode and execute the AR mode based on identifying a location (713) outside the reference identification area (710). For example, the wearable device (103) may provide a real environment by executing the AR mode.

[0160] Referring to FIG. 7B, an example (702) is illustrated in which a movement detection area (720) and a reference detection area (725) are further set for an external object (520) together with a plurality of areas (e.g., a first identification area (402), a reference identification area (not shown), and a second identification area (not shown)) set for a wearable device (103). Referring to example (702), the content for the first identification area (402) may be applied substantially in the same manner as in FIG. 7A. The reference detection area (725) may include the movement detection area (720). In other words, the reference detection area (725) may have a larger size than the movement detection area (720). In addition, the reference detection area (725) and the movement detection area (720) may be included in the first identification area (402). In example (702), an example is shown in which the reference detection area (725) and the movement detection area (720) are formed in a circular shape. However, this is merely for convenience of explanation, and the embodiments of the present disclosure are not limited thereto. For example, the reference detection area (725) and the movement detection area (720) may be formed in a shape different from a circle.

[0161] Referring to example (702), the wearable device (103) can identify an external object (520) while displaying a virtual environment within a VR mode. For example, the wearable device (103) can identify that the external object (520) is located within the first identification area (402). Based on identifying the location (726) of the external object (520), the wearable device (103) can maintain display of a screen including the virtual environment within the VR mode. In one embodiment, based on identifying that the location (726) changes to a location (727) due to movement of the external object (520), the wearable device (103) can display a visual object for the external object (520). For example, the wearable device (103) may display the screen including the visual object for the external object based on identifying a location (727) outside the movement detection area (720) and within the reference detection area (725). For example, the visual object may include text for notifying the risk of loss. For example, the visual object may include a rendered image of an area of ​​the real environment including the external object or the real environment. In addition, the visual object may further include text for notifying the movement of the image and the external object. However, the embodiments of the present disclosure are not limited thereto. According to one embodiment, the wearable device (103) may execute an AR mode that changes from the VR mode based on identifying a change from the location (727) to the location (728) due to the movement of the external object (520). For example, the wearable device (103) may terminate the VR mode and execute the AR mode based on identifying a location (728) outside the reference detection area (725). For example, the wearable device (103) may provide a real environment by executing the AR mode.

[0162] According to one embodiment, the wearable device (103) can change the sizes of the movement detection area (720) and the reference detection area (725) depending on the movement speed of the external object (520). For example, when the movement speed of the external object (520) is fast, the wearable device (103) can set the sizes of the movement detection area (720) and the reference detection area (725) to be small in order to react more sensitively. Accordingly, the wearable device (103) can display the visual object and switch to the AR mode more quickly.

[0163] Referring to FIG. 7c, an example (730) of a first identification area (402) and a second identification area (403) identified based on the direction of the FoV (730) of the wearable device (103) is illustrated. Referring to example (703), the contents of the first identification area (402) and the second identification area (403) can be applied substantially identically to FIG. 7a.

[0164] Referring to example (703), the shapes of the first identification area (402) and the second identification area (403) can be identified based on the direction in which the FoV (730) faces. According to one embodiment, the first identification area (402) and the second identification area (403) can be formed to be wider as the direction in which the FoV (730) faces increases. For example, a portion (732) of the first identification area (402) in a different direction (e.g., an opposite direction) from the direction in which the FoV (730) faces can be formed to be narrower than the remaining portion of the first identification area (402) toward which the direction faces. In addition, for example, a portion (733) of the second identification area (403) in a different direction (e.g., an opposite direction) from the direction in which the FoV (730) faces can be formed to be narrower than the remaining portion of the second identification area (403) toward which the direction faces. However, the embodiments of the present disclosure are not limited thereto. According to one embodiment, the first identification area (402) used in VR mode may have the same size regardless of the direction of the FoV (730). Also, although not shown in example (703), according to one embodiment, the reference identification area (710) may also be set based on the FoV (730).

[0165] Referring to FIG. 7d, an example (704) is illustrated in which a first identification area (742), a reference identification area (750), and a second identification area (743) are set for an external electronic device (530). Referring to example (704), the contents of the first identification area (742), the reference identification area (750), and the second identification area (743) can be applied substantially identically to the contents of the first identification area (402), the reference identification area (710), and the second identification area (403) of FIG. 7a.

[0166] Referring to example (704), the external electronic device (530) may set a first identification area (742) and a reference identification area (750) based on receiving a signal indicating that the wearable device (103) operates in VR mode. For example, the signal may include information (e.g., size) about the first identification area (742) and the reference identification area (750). The first identification area (742) and the reference identification area (750) may be included in a second identification area (743) for the AR mode. According to one embodiment, the first identification area (742) and the reference identification area (750) may be identified based on a distance between the wearable device (103) and the external electronic device (530). For example, when the distance is a first distance, the first identification area (742) and the reference identification area (750) may be set to a first size and a second size, respectively. When the distance is a second distance that is longer than the first distance, the first identification area (742) and the reference identification area (750) may be set to a third size and a fourth size, respectively. The third size and the fourth size may be formed narrower than the first size and the second size. This may be to more sensitively identify loss detection as the distance between the wearable device (103) and the external electronic device (530) increases. In addition, although not illustrated in example (704), according to one embodiment, the movement detection area (720) and the reference detection area (725) of example (702) may be set. For example, the movement detection area (720) and the reference detection area (725) may be changed based on the distance. Although not illustrated in example (704) of FIG. 7d, the external electronic device (530) may be positioned in a direction (or gaze direction) from which a user wearing the wearable device (103) is viewed. However, the embodiments of the present disclosure are not limited thereto.For example, an external electronic device (530) may be connected to a wearable device (103), but may be located in an area that cannot be visually observed by the user of the wearable device (103).

[0167] Figure 8 illustrates an example of a wearable device displaying a visual object to alert of a risk of loss of an external object.

[0168] Referring to FIG. 8, examples (800, 805) are shown of displaying a visual object for the external object based on the wearable device (103) identifying that the location of the external object is outside the first identification area and within the reference identification area.

[0169] Referring to examples (800) and (805), the wearable device (103) can display a screen including a virtual environment (810). For example, the wearable device (103) can display the screen including the virtual environment (810) through the display (503) while running a VR mode.

[0170] Referring to example (800), the wearable device (103) may display a visual object (820) for the external object based on identifying that the location is outside the first identification area and within the reference identification area. According to one embodiment, the visual object (820) may at least partially overlap with the virtual environment (810). For example, the visual object (820) may be displayed in a floating state on the virtual environment (810). For example, the visual object (820) may include text to notify of the risk of loss. For example, the text may include “risk of losing your smartphone.” In the example of FIG. 8, the external object is illustrated as a smartphone, but embodiments of the present disclosure are not limited thereto. For example, the external object may include a non-electronic device, such as a bag.

[0171] Referring to example (805), the wearable device (103) may display a visual object (830) for the external object based on determining that the location is outside the first identification area and within the reference identification area. According to one embodiment, the visual object (830) may at least partially overlap with the virtual environment (810). For example, the visual object (830) may include a rendered image representing the external object. Additionally, the visual object (830) may further include text for notifying the image and movement of the external object. For example, the image may represent a pre-stored smartphone, and the text may include "Move Smartphone." However, the embodiments of the present disclosure are not limited thereto. For example, the external object may include a non-electronic device, such as a bag.

[0172] A user of a wearable device (103) can recognize the possibility of loss of the external object, a smartphone, based on the visual object (820) and the visual object (830).

[0173] Although not shown in FIG. 8, in one embodiment, the wearable device (103) may provide other information instead of displaying a visual object (820) (or a visual object (830)) for the external object based on identifying that the location is outside the first identification area and within the reference identification area. For example, the other information may include a sound output through an audio output device of the wearable device (103) or a vibration based on a motor of the wearable device (103). In addition, in one embodiment, the wearable device (103) may provide both display of a visual object (820) (or a visual object (830)) for the external object and the other information based on identifying that the location is outside the first identification area and within the reference identification area.

[0174] According to one embodiment, the wearable device (103) may change settings for displaying the visual object (820) (or the visual object (830)). For example, the settings may include the font of the text included in the visual object (820) (or the visual object (830)), the color of the visual object (820) (or the visual object (830)), and the background color other than the visual object (820) (or the visual object (830)). For example, the wearable device (103) may change the settings depending on the degree of risk of loss. The degree of risk of loss may be identified based on the moving speed of the external object or the priority (or importance) set for each external object.

[0175] Figure 9 illustrates an example of a wearable device changing its operating mode.

[0176] Referring to FIG. 9, examples (900, 905) are shown of executing the AR mode changed from the VR mode based on the wearable device (103) identifying that the location of the external object is located outside the reference identification area.

[0177] Referring to example (900), the wearable device (103) may display a screen including a virtual environment (910) while the VR mode is running. For example, the wearable device (103) may display the screen including the virtual environment (910) through the display (503) based on identifying that the location of the external object is within the first identification area.

[0178] Referring to example (905), the wearable device (103) may terminate the VR mode and execute the AR mode based on identifying that the location is outside the reference identification area. For example, the wearable device (103) may provide a real environment (920) by executing the AR mode. For example, the real environment (920) may be provided through the display (503) of the wearable device (103). For example, the real environment (920) may be provided by being transmitted through the display (503). For example, the user of the wearable device (103) may identify a smartphone (930) within the real environment (920). Accordingly, the user may recognize that there is a higher risk of loss than the possibility of loss of the smartphone (930), which is an external object.

[0179] In one embodiment, the wearable device (103) can adjust the timing for switching from the VR mode to the AR mode. For example, the wearable device (103) can change the setting depending on the degree of risk of loss. The degree of risk of loss can be identified based on the moving speed of the external object, the priority (or importance) set for each external object, or the complexity or risk of the surrounding area of ​​the wearable device (103).

[0180] FIG. 10 illustrates an example of an operational flow for how a wearable device changes a visual object or operational mode based on loss detection for an external object.

[0181] At least some of the methods of FIG. 10 may be performed by the wearable device (103) of FIG. 5. For example, at least some of the methods may be controlled by the processor (501) of the wearable device (103). In the embodiments described below, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0182] In operation (1010), the wearable device (103) can identify an external object that is a target of location tracking based on the communication circuit (509). The external object may represent a target of location tracking. For example, the wearable device (103) can identify the external object based on the communication circuit (509). At this time, the external object may be an electronic device. For example, the wearable device (103) can identify the external object that is connectable or connected based on the communication circuit (509). Alternatively, for example, the wearable device (103) can identify the external object designated by a user of the wearable device (103). For example, the user can designate the external object based on at least a portion of an input to the wearable device (103). For example, the external object may be specified by the user when the wearable device (103) executes VR mode.

[0183] For example, the external object, which is an electronic device, can establish a connection with a wearable device (103). The wearable device (103) can identify the location of the external object based on the connection. For example, the wearable device (103) can obtain location information through the connection with the external object within a specified distance, and can identify the location of the external object based on the location information. Alternatively, the external object, which is an electronic device, can transmit and receive signals with the wearable device (103) while not establishing a connection with the wearable device (103). For example, the wearable device (103) can transmit a request signal to the external object and receive a response signal from the external object. Based on the request signal and the response signal, the wearable device (103) can identify the location of the external object. Location tracking performed within the established connection state can be referred to as a device finding service based on a connection method. Additionally, location tracking performed within a connection-unestablished state may be referred to as a device finding service based on a non-connection method. Although not described in the above examples, location tracking may further include location tracking that establishes and performs a connection only when necessary.

[0184] For example, the external object, which is a non-electronic device, can be identified by the wearable device (103). For example, the wearable device (103) can identify the external object designated by the user based on at least one of the sensor (505), the camera (507), or the communication circuit (509). For example, the wearable device (103) can identify the location of the external object based on information obtained using at least one of the sensor (505), the camera (507), or the communication circuit (509).

[0185] In operation (1020), the wearable device (103) may display a screen including a virtual environment through the display (509) within a VR mode that is executed based on identifying an event.

[0186] According to one embodiment, the wearable device (103) may execute a VR mode. For example, the wearable device (103) may execute the VR mode based on identifying the event for executing the VR mode. For example, the event may include at least one of an input to a physical button included in the wearable device (103) for executing the VR mode, an input to a portion of the display, or a gesture by a user of the wearable device (103) on the FoV. For example, the event may be referred to as an event for executing the VR mode, a mode conversion event, a mode change event, or a mode change input. For example, the physical button may include a button used for switching between an AR mode and a VR mode.

[0187] According to one embodiment, the wearable device (103) may identify a first identification area. For example, the wearable device (103) may identify the first identification area based on the execution of the VR mode. For example, the first identification area may be referred to as an identification area according to the VR mode, an identification area, a VR identification area, or a VR detection area.

[0188] According to one embodiment, the wearable device (103) may identify the first identification area that is changed from the second identification area of ​​the AR mode based on the execution of the VR mode that is changed from the AR mode. For example, the second identification area may be referred to as an identification area according to the AR mode, another identification area, an AR identification area, an AR detection area, or a basic detection area. For example, the second identification area may include the first identification area. In other words, the size of the second identification area may be wider than the size of the first identification area.

[0189] According to one embodiment, the first identification area may be identified based on at least one of the complexity of the surrounding area of ​​the wearable device (103), the risk of the surrounding area, the time at which the VR mode is executed, or the direction of the field of view (FoV) of the wearable device (103).

[0190] In operation (1030), the wearable device (103) can identify whether the location of the external object is within a reference identification area including an identification area for loss detection in the VR mode.

[0191] In one embodiment, the wearable device (103) can identify the location of the external object. For example, the wearable device (103) can identify whether the external object is moving.

[0192] According to one embodiment, the wearable device (103) can identify the distance between the wearable device (103) and an external object. For example, the wearable device (103) can identify the location of the external object based on the distance and the direction of the external object. For example, the wearable device (103) can identify the location based on information obtained from an access point (AP) to which the wearable device (103) and the external object are connected, information about a signal received by the wearable device (103) (e.g., a difference in the strength of a signal or the timing at which signals are received), or a location positioning technique. For example, the location positioning technique can include at least one of Wi-Fi-based location positioning, BLE-based location positioning, IoT (internet of things) server-based location positioning, UWB (ultra-wideband)-based location positioning, ToF (time of flight) camera-based location positioning, and GPS-based location positioning.

[0193] According to one embodiment, the wearable device (103) can identify whether the location of the external object is within a reference identification area. According to one embodiment, the reference identification area may, for example, represent an area used to identify a risk of loss within the VR mode.

[0194] In operation (1030), based on identifying that the location is within the reference identification area, the wearable device (103) may perform operation (1040). Alternatively, in operation (1030), based on identifying that the location is outside the reference identification area, the wearable device (103) may perform operation (1050).

[0195] In operation (1040), the wearable device (103) may display, through the display (503), the screen including a visual object for the external object based on identifying that the location is within the reference identification area. In operation (1050), the wearable device (103) may switch the VR mode to an AR (augmented reality) mode based on identifying that the location is outside the reference identification area.

[0196] In one embodiment, the wearable device (103) may display a visual object for the external object. For example, the wearable device (103) may display the screen including the visual object for the external object based on identifying that the location is outside the first identification area and within the reference identification area.

[0197] According to one embodiment, the wearable device (103) can track the external object. For example, the wearable device (103) can track the external object using an image acquired through a camera (507) included in the wearable device (103). The wearable device (103) can map between the external object identified through the communication circuit (509) and the external object tracked through the image. In other words, the wearable device (103) can identify the external object in the image based on the location identified through the communication circuit (509). The wearable device (103) can display the visual object in a display area within the display (503) of the wearable device (103) corresponding to the location of the external object in the image.

[0198] According to one embodiment, the visual object may at least partially overlap with the virtual environment provided within the VR mode. For example, the visual object may be displayed in a floating state on the virtual environment. For example, the visual object may include text for notifying the risk of loss. For example, the visual object may include a rendered image representing the external object. In addition, the visual object may further include text for notifying the movement of the image and the external object. However, the embodiments of the present disclosure are not limited thereto. For example, the wearable device (103) may also display a rendered image or the real environment for an area of ​​the real environment including the external object. For example, the rendered image or the real environment may at least partially overlap with the virtual environment.

[0199] Additionally, according to one embodiment, the wearable device (103) may provide the user with a notification different from the visual object. For example, the wearable device (103) may notify the user of a risk of loss by outputting auditory information (e.g., sound). Alternatively, for example, the wearable device (103) may notify the user of a risk of loss by outputting tactile information (e.g., vibration). Alternatively, for example, the wearable device (103) may provide at least one of the auditory information and / or the tactile information together with the visual object.

[0200] According to one embodiment, the wearable device (103) may change settings for displaying the visual object. For example, the settings may include the font of the text included in the visual object, the color of the visual object, and the background color of the visual object. For example, the wearable device (103) may change the settings depending on the degree of risk of loss. The degree of risk of loss may be identified based on the movement speed of the external object or the priority (or importance) set for each external object.

[0201] According to one embodiment, the wearable device (103) may execute the AR mode. For example, the wearable device (103) may change from the VR mode to the AR mode based on identifying that the location is outside the reference identification area. For example, the wearable device (103) may terminate the VR mode and execute the AR mode based on identifying that the location is outside the reference identification area. For example, the wearable device (103) may provide a real environment by executing the AR mode. For example, the real environment may be provided through the display of the wearable device (103). For example, the real environment may be provided by being transmitted through the display.

[0202] In one embodiment, the wearable device (103) can adjust the timing for switching from the VR mode to the AR mode. For example, the wearable device (103) can change the setting depending on the degree of risk of loss. The degree of risk of loss can be identified based on the moving speed of the external object, the priority (or importance) set for each external object, or the complexity or risk of the surrounding area of ​​the wearable device (103).

[0203] Referring to FIGS. 1 to 10 , an electronic device and method according to embodiments of the present disclosure provide a technology for controlling a loss detection (or location tracking) operation for an external object according to the operation mode of the wearable device (103) capable of providing extended reality. For example, the external object may include an object representing an electronic device or a non-electronic device. The electronic device and method according to embodiments of the present disclosure can identify the location of a designated external object and identify a risk of loss even while the user wearing the wearable device (103) is using a VR mode. The electronic device and method according to embodiments of the present disclosure can change a visual object or operation mode according to the identified degree of risk of loss. Accordingly, the electronic device and method according to embodiments of the present disclosure can provide information on the possibility of loss or whether the device has been lost in a situation where the user has difficulty checking the external environment. In addition, the electronic device and method according to embodiments of the present disclosure can improve the usability of the wearable device (103).

[0204] As described above, the wearable device (103) may include a memory (511) that stores instructions and includes one or more storage media. The wearable device (103) may include a display (503). The wearable device (103) may include a communication circuit (509). The wearable device (103) may include at least one processor (501). The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify an external object that is a target of position tracking based on the communication circuit (509). The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to display, through the display (503), a screen including a virtual environment within a VR (virtual reality) mode that is executed based on identifying an event. The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify whether the location of the external object is within a reference identification area including an identification area for loss detection in the VR mode. The above instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to display, through the display (503), the screen including a visual object for the external object based on identifying that the location is within the reference identification area.The above instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to switch the VR mode to an augmented reality (AR) mode based on identifying that the location is outside the reference identification area.

[0205] According to one embodiment, the external object may include an electronic device that provides information for location tracking in a state in which a connection is established with the wearable device (103), or an electronic device that provides information for location tracking by transmitting and receiving signals in a state in which a connection is not established with the wearable device (103).

[0206] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to execute the VR mode based on acquiring the event. The event may include at least one of an input to a physical button included in the wearable device (103) for executing the VR mode, an input to a portion of the display (503), or a gesture by a user of the wearable device (103) on a field of view (FoV) of the wearable device (103).

[0207] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to execute the AR mode based on identifying that a user of the wearable device (103) is wearing the wearable device (103). The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to execute the VR mode that switches from the AR mode based on identifying the event.

[0208] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify the identification area for loss detection in the VR mode based on the execution of the VR mode. The identification area for loss detection in the VR mode may be included in another identification area for loss detection in the AR mode. The other identification area may include the reference identification area.

[0209] According to one embodiment, the identification area may be identified based on at least one of the complexity of the surrounding area of ​​the wearable device (103), the risk of the surrounding area, the time at which the VR mode is executed, or the direction of the field of view (FoV) of the wearable device (103).

[0210] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify the location of the external object based on the communication circuit (509). Within the AR mode, the location may be identified based on a positioning technique. Within the VR mode, the location may be identified further based on at least one of information obtained from an access point (AP) to which the external object and the wearable device (103) are connected, or information about a signal received from the external object, together with the positioning technique.

[0211] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify whether the location is within the identification area. The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify whether the location is within the reference identification area based on identifying that the location is outside the identification area. The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify whether the location is within the reference identification area based on identifying that the location is within the identification area. The movement detection area may be used to identify a missing detection of the external object and may be included in the identification area.

[0212] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify whether the location of the external object is within a reference detection area that includes the movement detection area, based on identifying that the location is outside the movement detection area. The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify a new location of the external object, based on identifying that the location is within the movement detection area.

[0213] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to display, through the display (503), the visual object for the external object based on identifying that the location is within the reference sensing area. The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to execute the AR mode that switches from the VR mode based on identifying that the location is outside the reference sensing area.

[0214] In one embodiment, the visual object may include text to indicate a risk of loss of the external object or a rendered image representing the external object.

[0215] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to connect with an external electronic device that performs the location tracking of the external object via the communication circuit (509). The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to transmit a signal to the external electronic device, in response to the execution of the VR mode, notifying the execution of the VR mode. The signal may include information about the identification area and the reference identification area.

[0216] In one embodiment, the instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to receive another signal from the external electronic device indicating that the location of the external object is outside the identification area. The instructions, when individually or collectively executed by the at least one processor (501), may cause the wearable device (103) to identify, based on the another signal, whether the location is within the reference identification area.

[0217] According to one embodiment, the identification area and the reference identification area may be identified based on the location of the external electronic device. The identification area and the reference identification area may be related to a first distance between the external electronic device and the external object, or the first distance and a second distance between the wearable device (103) and the external electronic device.

[0218] As described above, a method performed by a wearable device (103) may include an operation of identifying an external object that is a target of position tracking. The method may include an operation of displaying a screen including a virtual environment within a virtual reality (VR) mode that is executed based on the identification of an event. The method may include an operation of identifying whether a location of the external object is within a reference identification area that includes an identification area for loss detection of the VR mode. The method may include an operation of displaying the screen including a visual object for the external object based on identifying that the location is within the reference identification area. The method may include an operation of switching the VR mode to an augmented reality (AR) mode based on identifying that the location is outside the reference identification area.

[0219] According to one embodiment, the external object may include an electronic device that provides information for location tracking in a state in which a connection is established with the wearable device (103), or an electronic device that provides information for location tracking by transmitting and receiving signals in a state in which a connection is not established with the wearable device (103).

[0220] According to one embodiment, the method may include an operation of executing the VR mode based on acquiring the event. The event may include at least one of an input to a physical button included in the wearable device (103) for executing the VR mode, an input to a portion of a display (503) of the wearable device (103), or a gesture of a user of the wearable device (103) on a field of view (FoV) of the wearable device (103).

[0221] In one embodiment, the method may include an operation of executing the AR mode based on identifying that a user of the wearable device (103) is wearing the wearable device (103). The method may include an operation of executing the VR mode that is switched from the AR mode based on identifying the event.

[0222] According to one embodiment, the method may include an operation of identifying the identification area for loss detection in the VR mode based on the execution of the VR mode. The identification area for loss detection in the VR mode may be included in another identification area for loss detection in the AR mode. The other identification area may include the reference identification area.

[0223] According to one embodiment, the identification area may be identified based on at least one of the complexity of the surrounding area of ​​the wearable device (103), the risk of the surrounding area, the time at which the VR mode is executed, or the direction of the field of view (FoV) of the wearable device (103).

[0224] The non-transitory computer-readable storage medium as described above may store one or more programs including instructions that, when individually or collectively executed by at least one processor (501) of a wearable device (103) including a display (503) and a communication circuit (509), cause the wearable device (103) to identify an external object that is a target of position tracking based on the communication circuit (509). The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to display a screen including a virtual environment through the display (503) in a virtual reality (VR) mode that is executed based on identifying an event. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to identify whether the location of the external object is within a reference identification area including an identification area for loss detection in the VR mode. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to display, through the display (503), the screen including a visual object for the external object based on identifying that the location is within the reference identification area.The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to switch the VR mode to an augmented reality (AR) mode based on identifying that the location is outside the reference identification area.

[0225] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0226] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0227] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0228] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0229] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0230] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (103), A memory (511) storing instructions and including one or more storage media; display (503); Communication circuit (509); and At least one processor (501) comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: Based on the above communication circuit (509), an external object that is a target of position tracking is identified; In a VR (virtual reality) mode that is executed based on identifying an event, a screen including a virtual environment is displayed through the display (503); Identifying whether the location of the above external object is within a reference identification area including an identification area for loss detection of the VR mode; Based on identifying that the location is within the reference identification area, displaying the screen including a visual object for the external object through the display (503); and Based on identifying that the above location is outside the above reference identification area, causing the VR mode to be switched to AR (augmented reality) mode. Wearable device (103).

2. In claim 1, The above external objects are: An electronic device that provides information for location tracking while a connection is established with the wearable device (103), or An electronic device that provides information for location tracking by transmitting and receiving signals within a state where the connection is not established with the wearable device (103), Wearable device (103).

3. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: Based on acquiring the above event, causing the VR mode to be executed, The above event includes at least one of an input to a physical button included in the wearable device (103) for executing the VR mode, an input to a portion of the display (503), or a gesture of a user of the wearable device (103) on the field of view (FoV) of the wearable device (103). Wearable device (103).

4. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: Based on identifying that a user of the wearable device (103) is wearing the wearable device (103), the AR mode is executed; Based on identifying the above event, causing the VR mode to be executed, which switches from the AR mode. Wearable device (103).

5. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: Based on the execution of the above VR mode, causing the identification area for loss detection of the above VR mode to be identified, The above identification area for loss detection in the above VR mode is included in another identification area for loss detection in the above AR mode, The above other identification area includes the above reference identification area, Wearable device (103).

6. In claim 1, The above identification area is identified based on at least one of the complexity of the surrounding area of ​​the wearable device (103), the risk of the surrounding area, the time at which the VR mode is executed, or the direction of the field of view (FoV) of the wearable device (103). Wearable device (103).

7. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: To cause the location of the external object to be identified based on the above communication circuit (509), Within the above AR mode, the location is identified based on a positioning technique, Within the VR mode, the location is identified based on at least one of information obtained from an AP (access point) to which the external object and the wearable device (103) are connected, or information about a signal received from the external object, together with the location determination technique. Wearable device (103).

8. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: Identify whether the above location is within the above identification area; Based on identifying that the location is outside the identification area, identifying whether the location is within the reference identification area; and Based on identifying that the location is within the identification area, causing identification of whether the location is within a movement detection area for the external object, The above movement detection area is used to identify the loss detection of the external object, and is included in the identification area. Wearable device (103).

9. In claim 8, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: Based on identifying that the location is outside the movement detection area, identifying whether the location of the external object is within a reference detection area that includes the movement detection area; Based on identifying that the above location is within the above movement detection area, causing a new location of the external object to be identified. Wearable device (103).

10. In claim 9, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: Based on identifying that the above location is within the above reference detection area, displaying the visual object for the external object through the display (503), and Causing to execute the AR mode, which switches from the VR mode, based on identifying that the location is outside the reference detection area. Wearable device (103).

11. In claim 1, The visual object may include text indicating a risk of loss of the external object or a rendered image representing the external object. Wearable device (103).

12. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: Connecting to an external electronic device that performs location tracking of the external object through the communication circuit (509); and In response to the execution of the VR mode, cause the external electronic device to transmit a signal notifying the execution of the VR mode, The signal comprises information about the identification area and the reference identification area. Wearable device (103).

13. In claim 12, The above instructions, when individually or collectively executed by the at least one processor (501), cause the wearable device (103) to: receiving another signal from said external electronic device indicating that said location of said external object is outside said identification area, and Based on said other signal, causing it to identify whether said location is within said reference identification area, Wearable device (103).

14. A method performed by a wearable device (103), wherein the method comprises: An action that identifies an external object that is the target of position tracking; An action of displaying a screen containing a virtual environment within a VR (virtual reality) mode that is executed based on identifying an event; An action of identifying whether the location of the external object is within a reference identification area including an identification area for loss detection in the VR mode; An action of displaying the screen including a visual object for the external object based on identifying that the location is within the reference identification area; and Based on identifying that the location is outside the reference identification area, the operation of switching the VR mode to AR (augmented reality) mode is included. method.

15. A non-transitory computer-readable storage medium, when individually or collectively executed by at least one processor (501) of a wearable device (103) including a display (503) and a communication circuit (509), causes the wearable device (103) to: Based on the above communication circuit (509), an external object that is a target of position tracking is identified; In a VR (virtual reality) mode that is executed based on identifying an event, a screen including a virtual environment is displayed through the display (503); Identifying whether the location of the above external object is within a reference identification area including an identification area for loss detection of the VR mode; Based on identifying that the location is within the reference identification area, displaying the screen including a visual object for the external object through the display (503); and storing one or more programs including instructions that cause the VR mode to be switched to an augmented reality (AR) mode based on identifying that the location is outside the reference identification area; A non-transitory computer-readable storage medium.

Images