Wearable device, method, and computer-readable storage medium for displaying one or more virtual objects on basis of external object

Abstract

According to an embodiment of the present invention, a wearable device may comprise: a camera; a display; a memory for storing instructions; and a processor. The instructions, when executed by the processor, may cause the wearable device to define a region including a plurality of areas on the basis of scanning a physical space. The instructions, when executed by the processor, may cause the wearable device to display a first virtual object related to an object in a first area via the display on the basis of the first area identified via the camera among the plurality of areas. The instructions, when executed by the processor, may cause the wearable device to display a second virtual object related to the object via the display on the basis of the first area identified via the camera after the first virtual object has been displayed.

Description

Wearable device, method, and computer-readable storage medium for displaying one or more virtual objects based on external objects

[0001] The various embodiments described below relate to a wearable device, a method, and a computer-readable storage medium for displaying one or more virtual objects based on an external object.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] In one embodiment, a wearable device may include a camera, a display, a memory storing instructions, and a processor. The instructions, when executed by the processor, may cause the wearable device to define a zone including a plurality of regions based on scanning of a physical space. The instructions, when executed by the processor, may cause the wearable device to display, on the display, a first screen including a first virtual object associated with an object within a first region identified by the camera among the plurality of regions. The instructions, when executed by the processor, may cause the wearable device to display, after displaying the first screen, a second screen associated with a second region identified by the camera among the plurality of regions through the display. The instructions, when executed by the processor, may cause the wearable device to display a third screen, including a second virtual object related to the object, through the display, based on the first area identified through the camera, after displaying the second screen.

[0005] In one embodiment, a method of a wearable device may include an operation of defining a zone including a plurality of areas based on scanning of an actual space. The method may include an operation of displaying a first screen including a first virtual object related to an object in a first area, based on a first area identified through a camera of the wearable device among the plurality of areas, through a display of the wearable device. The operation may include an operation of, after displaying the first screen, displaying a second screen related to a second area, based on a second area identified through the camera among the plurality of areas, through the display. The operation may include an operation of, after displaying the second screen, displaying a third screen including a second virtual object related to the object, based on the first area identified through the camera, through the display.

[0006] In one embodiment, a non-transitory computer readable storage medium storing one or more programs may include instructions that, when executed by a processor of a wearable device having a camera and a display, cause the wearable device to define an area including a plurality of areas based on scanning of a physical space. The one or more programs may include instructions that, when executed by the processor of the wearable device, cause the wearable device to display, through the display, a first screen including a first virtual object associated with an object within a first area identified through the camera among the plurality of areas. The one or more programs may include instructions that, when executed by the processor of the wearable device, cause the wearable device to display a second screen related to a second region identified through the camera among the plurality of regions after displaying the first screen. The one or more programs may include instructions that, when executed by the processor of the wearable device, cause the wearable device to display a third screen including a second virtual object related to the object through the display, based on the first region identified through the camera after displaying the second screen.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment, according to one embodiment.

[0008] FIGS. 2A and 2B illustrate an example of a perspective view of a wearable device, according to one embodiment.

[0009] FIGS. 3A and 3B illustrate an example of an appearance of a wearable device according to one embodiment.

[0010] FIG. 4 illustrates an example block diagram of a wearable device according to one embodiment.

[0011] FIG. 5 illustrates an example of a zone and multiple areas within an external space identified by a wearable device.

[0012] FIGS. 6A, 6B, 6C, 6D, 6E, and 6F illustrate exemplary states of a wearable device displaying a screen based on an external object within an area.

[0013] Figure 7 is a flow chart of an exemplary wearable device.

[0014] Figure 8 is a flowchart of an exemplary wearable device.

[0015] Figures 9a and 9b are flowcharts of an exemplary wearable device.

[0016] Figure 10 is an exemplary drawing for explaining the operation of a wearable device.

[0017] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0018] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0019] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0020] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0021] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0022] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0023] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0024] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0025] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0026] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0027] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0028] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0029] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0030] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0031] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0032] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0033] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0034] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0035] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0036] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0037] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0038] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0039] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101).

[0040] The electronic device (101) may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199). The electronic device (101) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0041] FIGS. 2A and 2B illustrate an example of a perspective view of a wearable device, according to one embodiment.

[0042] According to one embodiment, the wearable device (200) may have the form of glasses that are wearable on a body part of a user (e.g., head). The wearable device (200) of FIGS. 2A and 2B may be an example of the electronic device (101) of FIG. 1. The wearable device (200) may include a head-mounted display (HMD). For example, the housing of the wearable device (200) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (200) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0043] Referring to FIG. 2A, according to one embodiment, a wearable device (200) may include at least one display (250) and a frame (295) supporting at least one display (250).

[0044] According to one embodiment, the wearable device (200) can be worn on a part of a user's body. The wearable device (200) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (200). For example, the wearable device (200) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0045] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0046] Referring to FIG. 2B, at least one display (250) can provide the user with visual information transmitted from external light and other visual information distinct from the visual information through a lens included in the at least one display (250). The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When the user wears the wearable device (200), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0047] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (200) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0048] The wearable device (200) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (200) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (200) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (200) can view an image displayed on at least one display (250).

[0049] According to one embodiment, the frame (295) may be formed as a physical structure that allows the wearable device (200) to be worn on the user's body. According to one embodiment, the frame (295) may be configured so that, when the user wears the wearable device (200), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (295) may support at least one display (250). For example, the frame (295) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0050] Referring to FIG. 2A, the frame (295) may include a region (220) that at least partially contacts a part of the user's body when the user wears the wearable device (200). For example, the region (220) of the frame (295) that contacts a part of the user's body may include a region that contacts a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (200) makes contact with. According to one embodiment, the frame (295) may include a nose pad (210) that contacts a part of the user's body. When the wearable device (200) is worn by the user, the nose pad (210) may contact a part of the user's nose. The frame (295) may include a first temple (204) and a second temple (205) that contact a part of the user's body that is distinct from the part of the user's body.

[0051] For example, the frame (295) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of an ear of the wearer, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of an ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (200) can identify an external object (e.g., a user's fingertip) touching the frame (295) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (295).

[0052] According to one embodiment, the wearable device (200) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within a frame (295).

[0053] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (200) may be disposed on at least a portion of the frame (295) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B , but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B . When the number of microphones (265) included in the wearable device (200) is two or more, the wearable device (200) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (295).

[0054] In one embodiment, at least one optical device (282, 284) can project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) can be a projector. At least one optical device (282, 284) can be disposed adjacent to at least one display (250) or can be included within at least one display (250) as a part of at least one display (250). In one embodiment, the wearable device (200) can include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0055] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or motion recognition cameras (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (295) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (200). For example, the wearable device (200) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (200) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (200) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (200) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (200) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other. The wearable device (200) can obtain an image having visual quality of a first area matching the user's gaze and visual quality of a second area using foveated rendering.For example, if the wearable device (200) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1). An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0056] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (260-4) is superimposed on a virtual image provided through at least one optical device (282, 284). The wearable device (200) can compensate for depth information (e.g., the distance between the wearable device (200) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (260-4). The wearable device (200) can perform object recognition through an image acquired using a photographing camera (260-4). The wearable device (200) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the photographing camera (260-4). The wearable device (200) can perform a pass-through function to display an image acquired through the photographing camera (260-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the photographing camera (260-4) can be disposed on a bridge (203) disposed between a first rim (201) and a second rim (202).

[0057] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (200) and matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (200) looks straight ahead, the wearable device (200) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (200) is positioned.

[0058] The gesture recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The gesture recognition camera (260-2, 260-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (260-2, 260-3) can be used to perform simultaneous localization and mapping (SLAM) for 6 degrees of freedom pose (6 dof pose) and / or a spatial recognition function using a depth map. The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) may be positioned on the first rim (201) and / or the second rim (202).

[0059] The camera (260) included in the wearable device (200) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (200) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (200) can identify an external object based on a sensor for identifying the distance between the wearable device (200) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (200) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (200).

[0060] Although not shown, in one embodiment, the wearable device (200) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (295) and the hinge units (206, 207).

[0061] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (200). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0062] The antenna module (275) can transmit signals or power to the outside of the wearable device (200), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0063] The speaker (255) can output an audio signal to the outside of the wearable device (200). The audio output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (200). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus adjacent to the user's right ear.

[0064] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (200) to the user. For example, when the wearable device (200) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0065] Referring to FIG. 2B, according to one embodiment, a wearable device (200) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (200) (e.g., hardwares illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (200) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0066] According to one embodiment, a wearable device (200) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (200) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (200) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (200) based on the IMU.

[0067] FIGS. 3A and 3B illustrate an example of an appearance of a wearable device according to one embodiment.

[0068] The wearable device (300) of FIGS. 3A and 3B may be an example of the electronic device (101) of FIG. 1 and the wearable device (200) of FIGS. 2A and 2B. According to one embodiment, an example of the appearance of a first side (310) of a housing of the wearable device (200) is illustrated in FIG. 3A, and an example of the appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B.

[0069] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (200) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (200) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (200) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0070] According to one embodiment, the wearable device (200) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (200) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (200) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (200) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar representing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (200).

[0071] Referring to FIG. 3B, a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor 330) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A to obtain information related to the external environment of the wearable device (200). For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0072] For example, using cameras (260-11, 260-12), the wearable device (200) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be placed on the second face (320) of the wearable device (200) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be placed on the second face (320) of the wearable device (200) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0073] According to one embodiment, the wearable device (200) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (200) and an external object. Using the depth sensor (330), the wearable device (200) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (200). Although not illustrated, a microphone may be disposed on the second face (320) of the wearable device (200) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0074] FIG. 4 illustrates an example block diagram of a wearable device according to one embodiment.

[0075] Referring to FIG. 4, a wearable device (200) according to one embodiment may include at least one of a processor (410), a memory (415), a display (420), a camera (425), a sensor (430), or a communication circuit (435). The processor (410), the memory (415), the display (420), the camera (425), the sensor (430), and the communication circuit (435) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (402). The type and / or number of hardware components included in the wearable device (200) is not limited to those illustrated in FIG. 4. For example, the wearable device (200) may include only some of the hardware components illustrated in FIG. 4. The elements within the memory described below (e.g., layers and / or modules) may be logically distinct, but are not limited thereto.

[0076] According to one embodiment, the processor (410) of the wearable device (200) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (410) may be one or more. For example, the processor (410) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0077] The memory (415) of the wearable device (200) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (410). The memory (415) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi media card (eMMC).

[0078] In one embodiment, the display (420) of the wearable device (200) can output visualized information to a user of the wearable device (200). For example, the display (420) can be controlled by a processor (410) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (420) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0079] In one embodiment, the camera (425) of the wearable device (200) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (425) may be arranged in the form of a two-dimensional array. The camera (425) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (425) may mean one (a) two-dimensional frame data acquired from the camera (425). For example, video data captured using the camera (425) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (425) according to a frame rate. The camera (425) may further include a flash light that is positioned toward the direction in which the camera (425) receives light and outputs light toward the direction.

[0080] According to one embodiment, the wearable device (200) may include a plurality of cameras, for example, cameras (425), arranged in different directions. A first camera among the plurality of cameras may be referred to as a motion recognition camera (e.g., motion recognition cameras 260-2 and 260-3 of FIG. 2B ), and a second camera may be referred to as a gaze tracking camera (e.g., gaze tracking camera 260-1 of FIG. 2B ). The wearable device (200) may identify a position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (200) may identify a direction of a gaze of a user wearing the wearable device (200) using an image acquired using the second camera. For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0081] According to one embodiment, a sensor (430) of a wearable device (200) may generate electrical information that may be processed by a processor (410) and / or a memory (415) of the wearable device (200) from non-electronic information related to the wearable device (200). The information may be referred to as sensor data. The sensor (430) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (200), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (200).

[0082] In one embodiment, the communication circuit (435) of the wearable device (200) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (200) and an external electronic device. The communication circuit (435) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (435) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0083] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by the processor (410) of the wearable device (200) may be stored in the memory (415) of the wearable device (200). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (200) and / or the processor (410) may perform at least one of the operations according to the embodiments described below when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (200) may mean that one or more instructions provided in the form of an application are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (200)). As an example, an application may include a program and / or a library related to a service provided to a user.

[0084] Referring to FIG. 4, programs installed in the wearable device (200) may be classified into one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (200) (e.g., the display (420), the camera (425), and / or the sensor (430)) may be classified within the hardware abstraction layer (480). The framework layer (450) may be referred to as an XR framework layer in that it includes one or more programs for providing an XR (extended reality) service. For example, FIG. 4 illustrates layers being divided within the memory (415), the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (415).

[0085] For example, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a position tracker (471), a space recognizer (472), a gesture tracker (473), and / or an eye tracker (474), a face tracker (475)) may be classified. Programs classified within the framework layer (450) may provide an executable API (application programming interface) based on other programs.

[0086] For example, within the application layer (440), programs designed to target users controlling wearable devices (200) may be classified. Examples of programs classified into the application layer (440) include, but are not limited to, an XR (extended reality) system UI (user interface) and / or an XR application (442). For example, programs (e.g., software applications) classified into the application layer (440) may call an API (application programming interface) to cause execution of functions supported by programs classified into the framework layer (450).

[0087] For example, the wearable device (200) may display one or more visual objects on the display (420) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (441). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (200) may provide a service for controlling functions available within a virtual space to the user based on the execution of the XR system UI (441).

[0088] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plug-in (444) are illustrated as being included within the XR system UI (441), but are not limited thereto. For example, the XR system UI (441) may cause execution of functions supported by the lightweight renderer (443) and / or the XR plug-in (444) included within the framework layer (450).

[0089] For example, the wearable device (200) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (200) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (444). The XR plugin (444) may be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0090] For example, the wearable device (200) may display a screen representing at least a portion of a virtual space on the display (420) based on the execution of the XR application (442). The XR plug-in (444-1) included in the XR application (442) may be referenced by the XR plug-in (444) of the XR system UI (441). Descriptions of the XR plug-in (444-1) that overlap with the description of the XR plug-in (444) may be omitted. The wearable device (200) may cause the execution of the virtual space manager (451) based on the execution of the XR application (442).

[0091] According to one embodiment, the wearable device (200) may provide a virtual space service based on the execution of the virtual space manager (451). For example, the virtual space manager (451) may include a platform (e.g., an Android platform) for supporting the virtual space service. The wearable device (200) may display the posture of a virtual object representing the user's posture rendered using data acquired through the sensor (430) based on the execution of the virtual space manager (451) on the display. The virtual space manager (451) may be referred to as a composition presentation manager (CPM).

[0092] For example, the virtual space manager (451) may include a runtime service (452). As an example, the runtime service (452) may be referred to as an OpenXR runtime module. The wearable device (200) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (200) based on the execution of the runtime service (452). As an example, the wearable device (200) may be used to perform rendering for a virtual space service to a user based on the execution of the runtime service (452). For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the execution of the runtime service (452).

[0093] For example, the virtual space manager (451) may include a pass-through manager (453). Based on the execution of the pass-through manager (453), the wearable device (200) may display a screen representing a virtual space on the display (420), while another screen representing an actual space acquired through the camera (425) may be superimposed on at least a portion of the screen.

[0094] For example, the virtual space manager (451) may include an input manager (454). Based on the execution of the input manager (454), the wearable device (200) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (470). The wearable device (200) may initiate execution of at least one of the functions of the wearable device (200) using the acquired data.

[0095] For example, the perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). From the perspective of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referenced as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0096] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data acquired from a sensor (430) (or a camera (425)). The one or more programs may include at least one of a position tracker (471), a space recognizer (472), a gesture tracker (473), an eye tracker (474), and / or a face tracker (475). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those illustrated in FIG. 4.

[0097] For example, the wearable device (200) can identify the pose of the wearable device (200) using the sensor (430) based on the execution of the position tracker (471). The wearable device (200) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (200) using data acquired using the camera (425) and the IMU based on the execution of the position tracker (471). The position tracker (471) can be referred to as a head tracking (HeT) module.

[0098] For example, the wearable device (200) may be used to construct a three-dimensional virtual space around the wearable device (200) (or a user of the wearable device (200)) based on the execution of the space recognizer (472). The wearable device (200) may reconstruct the three-dimensional surroundings of the wearable device (200) using data acquired using the camera (425) based on the execution of the space recognizer (472). The wearable device (200) may identify at least one of a plane, a slope, and stairs based on the three-dimensionally reconstructed surroundings of the wearable device (200) based on the execution of the space recognizer (472). The space recognizer (472) may be referred to as a scene understanding (SU) module.

[0099] For example, the wearable device (200) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (200) based on the execution of the gesture tracker (473). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (430) based on the execution of the gesture tracker (473). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using a camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0100] For example, the wearable device (200) may identify (or track) eye movements of a user of the wearable device (200) based on the execution of the gaze tracker (474). As an example, the wearable device (200) may identify eye movements of the user using data acquired from at least one sensor based on the execution of the gaze tracker (474). As an example, the wearable device (200) may identify eye movements of the user based on data acquired using a camera (e.g., the gaze tracking camera (260-1) of FIGS. 2A and 2B) and / or an infrared light emitting diode (IR LED) based on the execution of the gaze tracker (474). The gaze tracker (474) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0101] For example, the recognition service layer (470) of the wearable device (200) may further include a face tracker (475) for tracking the user's face. For example, the wearable device (200) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (475). The wearable device (200) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (475). As an example, the wearable device (200) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (475).

[0102] For the embodiments described below, the wearable device (200) of FIG. 4 may be referenced. For example, the embodiments described below may be performed by the processor (410) of the wearable device (200) of FIG. 4.

[0103] FIG. 5 illustrates an example of a zone and multiple areas within an external space identified by a wearable device.

[0104] The wearable device (200) of FIG. 5, the electronic device (101) and / or processor (120) of FIG. 1, the wearable device (200) of FIG. 2A, the wearable device (300) of FIG. 3A, the wearable device (200) and / or processor (410) of FIG. 4 can perform the operation of the wearable device (200) of FIG. 5.

[0105] According to one embodiment, the wearable device (200) can perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when the user (50) wears the wearable device (200), the wearable device (200) can include at least one lens positioned adjacent to the user's (50) eyes. The wearable device (200) can combine ambient light passing through the lens with light emitted from a display of the wearable device (200). A display area of ​​the display can be formed within the lens through which the ambient light passes. Since the wearable device (200) combines the ambient light and the light emitted from the display, the user (50) can see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0106] In one embodiment, the wearable device (200) can perform functions related to video see-through (VST) and / or virtual reality (VR). For example, when a user (50) wears the wearable device (200), the wearable device (200) can include a housing that covers the eyes of the user (50). In the state, the wearable device (200) can include a display disposed on a first side of the housing facing the eyes. The wearable device (200) can include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (200) can acquire images and / or videos representing ambient light. The wearable device (200) can output the image and / or video within the display disposed on the first surface, thereby allowing the user (50) to recognize the ambient light through the display. The displaying area (or displaying region) (or active area or active region) of the display disposed on the first surface can be formed by one or more pixels included in the display. The wearable device (200) can synthesize a virtual object into the image and / or video output through the display, thereby allowing the user (50) to recognize the virtual object together with an actual object recognized by the ambient light.

[0107] According to one embodiment, the wearable device (200) can identify or recognize a position (or location) and / or direction (or orientation) of the wearable device (200) based on images (and / or videos) obtained or acquired using a camera. Referring to FIG. 5, within an external space (e.g., a room) including external objects (531, 532, 533, 534, 535, 536), the wearable device (200) can obtain spatial information about the external space using one or more cameras and / or at least one sensor. The spatial information can include a geographic location (e.g., global positioning system (GPS) coordinates) of the external space identified from at least one sensor. The spatial information can include images and / or videos of the external space identified from one or more cameras. The wearable device (200) can perform object recognition on an image and / or video to identify external objects (531, 532, 533, 534, 535, 536) included in the external space from the image and / or video.

[0108] The wearable device (200) may provide a user experience based on virtual reality and / or augmented reality while being worn by a user (50). For example, the wearable device (200) may display a virtual object or an imaginary object that covers at least a portion of the field-of-view (FoV) of the user (50). In one embodiment, the wearable device (200) may identify a region (510) for providing virtual reality and / or augmented reality within an external space based on the distribution (or positional relationship) of real objects (e.g., external objects (531, 532, 533, 534, 535, 536)) within the external space.

[0109] According to one embodiment, the wearable device (200) may divide the zone (510) into one or more areas. Referring to FIG. 5, exemplary multiple zones (520) within the zone (510) defined by the wearable device (200) are illustrated. The wearable device (200) may determine the multiple zones (520) within the zone (510) using real objects included within the zone (510). For example, the wearable device (200) may determine the multiple zones (520) through the shape, position, arrangement relationship, etc. of the multiple real objects within the zone (510) identified from one or more cameras. For example, the wearable device (200) may determine a portion of the zone (510) including external objects (531, 532) as the first zone (521). For example, the wearable device (200) may determine a portion of the zone (510) including external objects (533, 534) as the second zone (522). For example, the wearable device (200) may determine a portion of the zone (510) including external objects (535, 536) as the third zone (523). The above-described embodiments are exemplary, and the wearable device (200) may determine or define one or more zones within the zone (510) by using spatial information of the zone (510) acquired through one or more cameras and / or at least one sensor.

[0110] According to one embodiment, the wearable device (200) may provide virtual reality and / or augmented reality to a user (50) wearing the wearable device (200) by using a zone (510) and / or a plurality of zones (520). For example, based on one zone identified from one or more cameras of the wearable device (200) among the plurality of zones (520), the wearable device (200) may provide a virtual reality related to the one zone and / or a virtual object related to an external object included in the one zone to the user (50) through a display of the wearable device (200). For example, the wearable device (200) may identify a first zone (521) within the zone (510) based on identifying an external object (531) and an external object (532) within the zone (510) and / or a placement relationship between the external objects (531, 532). The wearable device (200) may, based on identifying the first region (521), provide virtual reality and / or augmented reality related to the first region (521) or provide virtual objects related to external objects (531, 532) within the first region (521) to the user through a display. The embodiment is not limited thereto, and the wearable device (200) may, based on spatial information, identify one of a plurality of regions (520) included within the zone (510). Based on identifying the one region within the zone (510), the wearable device (200) may output a screen, an image, one or more virtual objects, and / or audio based on virtual reality and / or augmented reality to the user (50).

[0111] According to the above-described embodiment, the wearable device (200) may be configured to determine a zone (510) including a plurality of areas (520) based on scanning of an external space. The wearable device (200) may provide virtual reality and / or augmented reality related to one area, or provide a virtual object related to an external object within the one area through a display, based on one area identified among the plurality of areas (520).

[0112] Hereinafter, exemplary operations of a wearable device for providing virtual reality and / or augmented reality for a plurality of areas (520) within a zone (510) are described with reference to FIGS. 6A to 6F.

[0113] FIGS. 6A, 6B, 6C, 6D, 6E, and 6F illustrate exemplary states of a wearable device displaying a screen based on an external object within an area.

[0114] The operations of the wearable device (200) of FIGS. 6A, 6B, 6C, 6D, 6E, and 6F may be performed by the electronic device (101) and / or the processor (120) of FIG. 1, the wearable device (200) of FIG. 2A, the wearable device (300) of FIG. 3A, the wearable device (200) and / or the processor (410) of FIG. 4.

[0115] According to one embodiment, within an exemplary external space including external objects (531, 532, 533, 534, 535, 536), the wearable device (200) may determine, based on a space scan, a zone (510) and / or a plurality of zones (520) within the zone (510). The plurality of zones (520) may be related to a positional relationship of the external objects (531, 532, 533, 534, 535, 536) within the external space, a position of the wearable device (200) within the external space (e.g., an elevation of the wearable device (200) from the floor of the external space), and / or a posture of a user (50) wearing the wearable device (200).

[0116] According to one embodiment, the wearable device (200) may receive data related to virtual reality and / or augmented reality from an external electronic device (e.g., the electronic device (102) of FIG. 1 or the external electronic device (1100) of FIG. 10) or a server (e.g., the server (108) of FIG. 1) to provide virtual reality and / or augmented reality related to one of the plurality of regions (520) based on scanning of the region. For example, the wearable device (200) may be configured to store data of maps and / or virtual objects related to virtual reality and / or augmented reality received from the external electronic device (1100) of FIG. 10 in a memory (e.g., the memory (415) of FIG. 4). The wearable device (200) may be configured to display a map related to virtual reality and / or augmented reality related to one of the plurality of areas (520) through a display, or to display a virtual object related to an external object included in the one area within the map, based on identifying one area among the plurality of areas (520).

[0117] For example, the wearable device (200) may update in real time a map related to a plurality of areas (520) displayed through the display or virtual objects related to external objects (531, 532, 533, 534, 535, 536) included in the plurality of areas (520) based on spatial information of an area (510), a location of the wearable device (200) within the area (510), and / or a location of a user (50) identified by one or more cameras and / or at least one sensor of the wearable device (200). For example, as the area being scanned changes depending on the location of the user, the wearable device (200) may change or newly display a map related to the area displayed through the display or a virtual object related to an external object within the area.

[0118] Referring to FIG. 6A, an exemplary state (601) in which a first area (521) is scanned by a wearable device (200) worn by a user (50) is illustrated. The wearable device (200) may identify the first area (521) based on the position or posture of the user wearing the wearable device (200) (e.g., posture in the direction (-x direction) toward the first area (521)) and / or spatial information of the first area (521) (e.g., arrangement relationship between external objects (531, 532)) while the user moves within the area (510) via a path (611). The wearable device (200) can display virtual reality and / or augmented reality related to the first area (521) through a display within a state in which the first area (521) has been identified, or can display virtual objects related to external objects (531, 532) within the first area (521).

[0119] For example, in a state (601) in which a virtual reality related to a first area (521) is provided based on a screen (621), the wearable device (200) may provide a virtual image and / or video related to the first area (521). For example, the wearable device (200) may use spatial information of the first area (521) to display a virtual object (631) corresponding to an external object (531) within the first area (521) and a virtual object (632) corresponding to an external object (532) within the screen (621). For example, the wearable device (200) may display virtual objects (631, 632) together with a camera image based on a field-of-view (FOV) of the wearable device (200) corresponding to the first area (521).

[0120] According to one embodiment, while a screen (621) related to a first region (521) is displayed, the wearable device (200) may be configured to identify whether a time for which the first region (521) is identified is less than a reference time. Based on identifying the time as being greater than or equal to the reference time, the wearable device (200) may be configured to display a visual object (640) for determining whether to re-display the virtual reality or virtual objects (631, 632) provided through the screen (621) if the first region (521) is re-identified after the screen (621) is displayed.

[0121] For example, referring to FIG. 6E together, the wearable device (200) can identify the first area (521) through a path (611) and another path (614). The wearable device (200) can display a screen (625) providing the different virtual reality based on a user input through a visual object (642) for displaying a different virtual reality than the virtual reality of the screen (621) provided through the path (611). For example, while a user moves within a zone (510) via a path (614), within a state (604) in which a first zone (521) is identified, the wearable device (200) can display, through a screen (625), a virtual object (635) corresponding to an external object (531) within the first zone (521) and different from the virtual object (631), and a virtual object (636) corresponding to an external object (532) and different from the virtual object (632).

[0122] For example, referring to FIG. 6F together, the wearable device (200) can identify a first area (521) through a path (611) and another path (614). The wearable device (200) can display a screen (626) providing the virtual reality based on a user input through a visual object (641) for displaying a virtual reality corresponding to the virtual reality of the screen (621) provided through the path (611). For example, while the user moves within the zone (510) through the path (614), in a state (601) in which the first area (521) is identified, the wearable device (200) can display again a virtual object (635) corresponding to an external object (531) within the first area (521) and a virtual object (636) corresponding to an external object (532) through the screen (626).

[0123] The above-mentioned embodiments are exemplary, and the wearable device (200) may provide another virtual reality and / or another virtual object based on the one area identified through another path without displaying a visual object (e.g., a visual object (640)) for determining whether to re-provide the virtual reality and / or virtual object provided through the one area based on the time at which the one area among the plurality of areas (520) is identified being greater than or equal to a reference time. The wearable device (200) may provide substantially the same virtual reality and / or virtual object based on the one area identified through another path based on the identification that the identified time is less than or equal to a reference time.

[0124] Referring to FIG. 6B, an exemplary state (602) is illustrated in which a second area (522) is scanned by a wearable device (200) worn by a user (50). The wearable device (200) may identify the first area (521) based on the position or posture of the user wearing the wearable device (200) (e.g., posture in the direction (-y direction) toward the second area (522)) and / or spatial information of the second area (522) (e.g., arrangement relationship between external objects (533, 534)) while the user moves within the area (510) via a path (612). The wearable device (200), within a state in which the first region (521) is identified, may display virtual reality and / or augmented reality related to the first region (521) through a display, or may display virtual objects related to external objects (533, 534) within the second region (522). For example, within a state (602) in which virtual reality related to the second region (522) is provided based on the screen (622), the wearable device (200) may provide virtual images and / or videos related to the first region (521). For example, the wearable device (200) may use spatial information of the second region (522) to display a virtual object (633) corresponding to an external object (533) within the second region (522), and a virtual object (634) corresponding to an external object (534).

[0125] Referring to FIG. 6C, an exemplary state (603) is illustrated in which a third area (523) is scanned by a wearable device (200) worn by a user (50). The wearable device (200) may identify the third area (523) based on the position or posture (e.g., sitting posture in the +x direction) of the user wearing the wearable device (200) and / or spatial information of the third area (523) (e.g., arrangement relationship between external objects (535, 536)) while the user moves within the area (510) via a path (613). The wearable device (200) can display virtual reality and / or augmented reality related to the third area (523) through a display within the state of identifying the third area (523) or display virtual objects related to external objects (535, 536) within the third area (523).

[0126] For example, in a state (603) in which augmented reality related to a third area (523) is provided based on a screen (623), the wearable device (200) may provide virtual images and / or videos related to the third area (523). The augmented reality may be provided to another wearable device (e.g., another wearable device (1000) of FIG. 10) worn by another user through a server (e.g., a server (108) of FIG. 1) and / or an external electronic device (e.g., an electronic device (102) of FIG. 1, an external electronic device (1100) of FIG. 10). For example, the wearable device (200) may be connected to another wearable device (1100) worn by another user (10) through the external electronic device (1100) of FIG. 10. The other wearable device (1000) can display augmented reality related to the third area (523) provided from the external electronic device (1100) through the display of the other wearable device (1000). The wearable device (200) can display a visual object (650) related to the other user (10) through the screen (623). The sharing of virtual reality and / or augmented reality by the wearable devices (200, 1000) through the external electronic device (1100) in FIG. 10 will be described later.

[0127] Referring to FIG. 6d, an exemplary state (601) is illustrated in which a first area (521) is scanned by a wearable device (200) worn by a user (50). While the user moves within the area (510) via a path (611) and a different path (614), the wearable device (200) may identify the first area (521) based on the position or posture of the user wearing the wearable device (200) (e.g., posture in the direction (-x direction) toward the first area (521)) and / or spatial information of the first area (521) (e.g., arrangement relationship between external objects (531, 532)). According to one embodiment, the wearable device (200) may perform an event to determine to display, through the display, a virtual reality and / or augmented reality that is substantially the same as the virtual reality and / or augmented reality that was provided to the screen (621) through the path (611), based on identifying spatial information of the first region (521) that has been previously acquired (e.g., information on the arrangement relationship of external objects (531, 532) within the first region (521) acquired through the path (611).

[0128] For example, within a state (601) in which a first region (521) is identified through a path (611) and another path (614), the wearable device (200) may be configured to bypass displaying virtual reality and / or augmented reality through the display. The wearable device (200) may, based on the identification of spatial information related to the first region (521) previously acquired, display a visual object (660) through the display to determine whether to display a virtual reality substantially identical to the virtual reality provided through the path (611).

[0129] For example, referring to FIG. 6E together, the wearable device (200) may display a screen (625) providing a different virtual reality based on a user input through a visual object (662) for displaying a different virtual reality than the virtual reality of the screen (621) provided through the path (611). For example, while the user moves within the zone (510) through the path (614), within a state (604) in which a first area (521) is identified, the wearable device (200) may display, through the screen (625), a virtual object (635) corresponding to an external object (531) within the first area (521) and different from the virtual object (631), and a virtual object (636) corresponding to the external object (532) and different from the virtual object (632).

[0130] For example, referring to FIG. 6F together, the wearable device (200) may display a screen (626) providing the virtual reality based on a user input through a visual object (661) for displaying a virtual reality corresponding to the virtual reality of the screen (621) provided through the path (611). For example, while the user moves within the zone (510) through the path (614), within a state (601) in which the first area (521) is identified, the wearable device (200) may display again a virtual object (635) corresponding to an external object (531) within the first area (521) and a virtual object (636) corresponding to an external object (532) through the screen (626).

[0131] According to the above-described embodiment, the wearable device (200) can provide virtual reality and / or augmented reality related to one identified area among the plurality of areas (520) by using spatial information about the plurality of areas (520) within the zone (510). The wearable device (200) can provide a variety of user experiences to the user of the wearable device (200) by configuring it to provide different virtual realities and / or augmented realities according to user input and / or settings for the one area (e.g., the first area (521)) identified through different paths.

[0132] Figure 7 is a flow chart of an exemplary wearable device.

[0133] The operations of FIG. 7 may be performed by the electronic device (101) and / or processor (120) of FIG. 1, the wearable device (200) of FIG. 2A, the wearable device (300) of FIG. 3A, the wearable device (200) executing the spatial recognizer (472) of FIG. 4, and / or the processor (410).

[0134] Referring to FIG. 7, in operation (701), a processor of a wearable device may recognize an external space. The processor may identify a shape of the external space based on an image and / or video acquired from a camera (e.g., a camera (425) of FIG. 4). Based on the image and / or video, the processor may identify at least one external object included in the external space (e.g., external objects (531, 532, 533, 534, 535, 536) of FIG. 6A). The processor may obtain spatial information of the external space or determine a position and / or direction of the wearable device using a sensor (e.g., a sensor (430) of FIG. 4) (e.g., an IMU).

[0135] In operation (703), the processor of the wearable device can identify whether spatial information corresponding to an external space is stored. The spatial information can be matched to the external space recognized based on operation (701). The spatial information can be stored in a memory of the wearable device (e.g., memory (415) of FIG. 4 ). The spatial information can include data representing one or more regions segmented from the external space (e.g., multiple regions (520) of FIG. 6A ). If spatial information is not identified from the memory of the wearable device (703-No), the processor can perform operation (705).

[0136] In operation (705), the processor of the wearable device may determine a zone for virtual reality and / or augmented reality (e.g., zone (510) of FIG. 6A) based on a distribution of external objects included in the external space. The external objects of operation (705) may be identified based on object recognition of images and / or videos output from a camera. For example, the processor may identify a plurality of segmented zones within a zone formed to provide virtual reality and / or augmented reality based on data from sensors and / or cameras. Based on operation (705), the processor of the wearable device may perform a scan of a physical space. Based on the scan, the processor may define a zone including a plurality of zones.

[0137] In operation (707), the processor of the wearable device may display a virtual object related to an external object through a display based on the segmented regions within the zone. The processor may display a virtual object related to an external object within one of the multiple segmented regions through the display based on identifying the region. Based on the virtual object, the processor may provide a user experience related to virtual reality and / or augmented reality.

[0138] Figure 8 is a flowchart of an exemplary wearable device.

[0139] The operations of FIG. 8 may be performed by the electronic device (101) and / or the processor (120) of FIG. 1, the wearable device (200) of FIG. 2A, the wearable device (300) of FIG. 3A, the wearable device (200) executing the spatial recognizer (472) of FIG. 4, and / or the processor (410). The operation (801) of FIG. 8 may correspond to the operation (701) of FIG. 7.

[0140] Referring to FIG. 8, in operation (803), the processor of the wearable device may display a first screen (e.g., screen (621) of FIG. 6A) including a first virtual object (e.g., virtual objects (631, 632) of FIG. 6A) related to an external object (e.g., external objects (531, 532) of FIG. 5) within the first area (521) identified through first spatial information.

[0141] In operation (805), the processor of the wearable device may identify whether the similarity between the acquired spatial information and the first spatial information is less than a reference value after the first spatial information is acquired. For example, the processor may identify whether the similarity between the acquired spatial information and the first spatial information is less than a reference value based on the shape or arrangement relationship of external objects within the external space. If it is not identified that the similarity is less than the reference value (805-No), the processor of the wearable device may perform operation (809).

[0142] In operation (807), a second area (e.g., a second area (522) of FIG. 5) different from the first area may be identified based on identifying that the similarity between the first spatial information and the spatial information acquired after the first spatial information is acquired is less than a reference value. In operation (807), the processor may display a second screen (e.g., a screen (622) of FIG. 6B) related to the second area through a display based on the identified second area. For example, the processor may display a virtual object (e.g., virtual objects (633, 634) of FIG. 6B) related to a virtual reality, an augmented reality, and / or an external object (e.g., external objects (533, 534) of FIG. 5) included in the second area through a display based on the identified second area.

[0143] In operation (809), the processor may display a third screen (e.g., screen (625) of FIG. 6E) including a second virtual object (e.g., virtual objects (635, 636) of FIG. 6E) related to an external object within the first area based on the identified first area. For example, in operation (809), the processor may display a third screen providing a different virtual reality and / or different augmented reality than the first screen based on the identified spatial information related to the first area.

[0144] In operation (811), the processor of the wearable device may determine whether spatial information of multiple areas within a zone has been acquired. For example, based on the identification of segmented areas within the zone, the processor may determine whether spatial information related to the segmented areas in a memory (e.g., memory (415) of FIG. 4) is greater than or equal to a reference value. If spatial information less than the reference value is identified (operation 811-No), the processor may perform operation (809).

[0145] In operation (813), the processor of the wearable device may display at least one of the plurality of screens that were displayed through the display before the second screen was displayed after the first screen was displayed, based on identifying that all spatial information of the plurality of areas within the zone has been acquired. For example, referring to FIGS. 6A and 6F together, the processor may identify that all of the plurality of segmented areas (520) within the zone (510) have been explored through paths (611, 612, 613, and 614). Based on the spatial information of the plurality of areas (520) and the first area (521) identified through path (614), the processor may re-display at least one of the plurality of screens that were provided through path (614) after a screen based on the first area (521) (e.g., screen (626) of FIG. 6F) is provided.

[0146] Figures 9a and 9b are flowcharts of an exemplary wearable device.

[0147] The operations of FIGS. 9A and 9B may be performed by the electronic device (101) and / or the processor (120) of FIG. 1, the wearable device (200) of FIG. 2A, the wearable device (300) of FIG. 3A, the wearable device (200) executing the spatial recognizer (472) of FIG. 4, and / or the processor (410). Operations (901), (903), and (905) may correspond to operations (901), (803), and (807) of FIG. 8, respectively.

[0148] Referring to FIG. 9A, at operation (907), the processor of the wearable device may perform an event to determine whether to display external objects (e.g., external objects (531, 532) of FIG. 5) within the first region (e.g., the first region (521) of FIG. 5) via the display based on the identified first region. For example, referring to FIG. 6D, based on the first region (521) identified via path (614), the processor may display a visual object (660) to determine whether to display again virtual objects (631, 632) associated with the external objects (531, 532) within the first region (521) that were provided via path (611) via the display.

[0149] Referring to FIG. 9B, in operation (915), the processor of the wearable device may measure the time at which the first area is identified through the camera. For example, the processor may measure the time at which the first area is scanned or detected through the camera (e.g., the camera (425) of FIG. 4) and / or the sensor (e.g., the sensor (430) of FIG. 4).

[0150] Referring to FIG. 9A, at operation (909), the processor of the wearable device may identify whether a user input for displaying a virtual object other than the first virtual object (e.g., virtual objects (635, 636) of FIG. 6E) has been received. If the user input for the other virtual object is not identified (909-No), the processor may perform operation (913).

[0151] Referring to FIG. 9B, in operation (917), the processor of the wearable device may determine whether the time for identifying the first region is less than the reference time. If it is not determined that the time for identifying the first region is less than the reference time (917-No), the processor may perform operation (913).

[0152] In operation (911), the processor of the wearable device may display a third screen (e.g., screen (625) of FIG. 6e) including a second virtual object (e.g., virtual objects (635, 636) of FIG. 6e) related to an external object within the first region based on receiving a user input for displaying a first virtual object and a different virtual object. For example, in FIG. 6d, based on a user input through a visual object (662) for displaying a virtual reality different from the virtual reality provided through the path (611) based on the identified first region (521), the processor may display a screen (625) providing the different virtual reality through the display.

[0153] In operation (913), the processor of the wearable device may display a fourth screen (e.g., screen (626) of FIG. 6F) including a first virtual object related to an external object within the first region through the display. For example, in FIG. 6D, based on a user input through a visual object (661) for displaying a virtual reality substantially identical to the virtual reality provided through the path (611) based on the identified first region (521), the processor may display a screen (626) providing the virtual reality through the display.

[0154] Figure 10 is an exemplary drawing for explaining the operation of a wearable device.

[0155] Referring to FIG. 10, a wearable device (200) worn by a user (50) and another wearable device (1000) worn by another user (1000) may be connected via an external electronic device (1100). According to one embodiment, the other wearable device (1000) may be configured substantially identically or similarly to the wearable device (200).

[0156] According to one embodiment, the wearable devices (200, 1000) may receive, in real time, data related to virtual reality and / or augmented reality that may be provided based on spatial information of an external space from an external electronic device (1100). For example, the wearable device (200) may receive data related to virtual reality and / or augmented reality applicable through an external space (e.g., data related to a virtual map and a virtual object) from the external electronic device (1100) while an application for providing virtual reality and / or augmented reality (e.g., XR application (442) of FIG. 4) is running, thereby providing a virtual reality and / or augmented reality related to the external space to a user (50) through a display of the wearable device (200). An external electronic device (1100) can identify a similarity between spatial information obtained from a wearable device (200) and spatial information obtained from another wearable device (1000) based on identifying another wearable device (1000) on which an application for virtual reality and / or augmented reality is executed. The external electronic device (1100) can connect the other wearable device (1000) to the wearable device (200) based on identifying that the similarity is greater than or equal to a reference value. The external electronic device (1100) can provide substantially the same virtual reality to each of the wearable devices (200, 1000) based on the connection of the wearable devices (200, 1000). For example, referring to FIG. 6c together, the external electronic device (1100) may provide substantially the same virtual reality or display a visual object (650) relevant to the user through each of the wearable devices (200, 1000) based on the connection of the wearable devices (200, 1000).

[0157] According to the above-described embodiment, a wearable device (e.g., electronic device (101) of FIG. 1, wearable device (200) of FIG. 2A, wearable device (300) of FIG. 3A) may include a camera (e.g., camera (260) of FIG. 2B, camera (425) of FIG. 4), a display (e.g., at least one display (250) of FIG. 2A), a memory for storing instructions (e.g., memory (130) of FIG. 1, memory (415) of FIG. 4), and a processor (e.g., processor (120) of FIG. 1, processor (410) of FIG. 4). The instructions, when executed by the processor, may cause the wearable device to define a zone (e.g., zone (510) of FIG. 5) including a plurality of zones (e.g., zones (520) of FIG. 5) based on scanning of a physical space. The instructions, when executed by the processor, may cause the wearable device to display, through the display, a first screen (e.g., screen (621) of FIG. 6A) including a first virtual object (e.g., virtual objects (631, 632) of FIG. 6A) associated with an object (e.g., external objects (531, 532) of FIG. 5) within a first zone (e.g., first zone (521) of FIG. 5) identified through the camera among the plurality of zones. The above instructions, when executed by the processor, may cause the wearable device to display, through the display, a second screen (e.g., screen (622) of FIG. 6b) related to a second area (e.g., second area (522) of FIG. 5) identified through the camera among the plurality of areas after displaying the first screen.The instructions, when executed by the processor, may cause the wearable device to display a third screen (e.g., screen (625) of FIG. 6e) including a second virtual object (e.g., virtual objects (635, 636) of FIG. 6e) related to the object, based on the first area identified through the camera, after displaying the second screen.

[0158] For example, the instructions, when executed by the processor, may cause the wearable device to identify the plurality of areas using spatial information related to the actual space acquired through the camera.

[0159] For example, the instructions, when executed by the processor, may cause the wearable device to display the first screen through the display based on the first region identified through the first spatial information. The instructions, when executed by the processor, may cause the wearable device to display the second screen through the display based on the second region identified based on obtaining second spatial information having a similarity less than a reference value with the first spatial information. The instructions, when executed by the processor, may cause the wearable device to display the third screen through the display based on the first region identified based on obtaining third spatial information having a similarity greater than or equal to a reference value with the first spatial information.

[0160] For example, the instructions, when executed by the processor, may cause the wearable device to perform an event for determining, after displaying the second screen, whether to display the first virtual object related to the object within the first area based on the first area identified through the camera through the display. The instructions, when executed by the processor, may cause the wearable device to, in response to a user input for displaying the second virtual object, display the third screen including the second virtual object through the display. The instructions, when executed by the processor, may cause the wearable device to, in response to a user input for displaying the first virtual object, display the fourth screen (e.g., screen (626) of FIG. 6F) including the first virtual object through the display.

[0161] For example, the instructions, when executed by the processor, may cause the wearable device to identify whether a time for which the first region is identified through the camera while the first screen including the first virtual object is displayed through the ㅖ is less than a reference time. The instructions, when executed by the processor, may cause the wearable device to display, through the display, the third screen including the second virtual object based on the first region identified after the second screen is displayed, based on the identification of the time being less than the reference time. The instructions, when executed by the processor, may cause the wearable device to display, through the display, the fourth screen including the first virtual object based on the first region identified after the second screen is displayed, based on the identification of the time being greater than or equal to the reference time.

[0162] For example, the instructions, when executed by the processor, may cause the wearable device to, after displaying the second screen, identify whether all of the spatial information of the plurality of areas within the zone has been acquired based on spatial information associated with the actual space acquired through the camera. The instructions, when executed by the processor, may cause the wearable device, in response to identifying that all of the spatial information has not been acquired, to display, through the display, the third screen including the second virtual object associated with the object within the first area. The instructions, when executed by the processor, may cause the wearable device, in response to identifying that all of the plurality of areas have been identified, to display, through the display, at least one of the plurality of screens that were displayed through the display after the first screen was displayed and before the second screen was displayed.

[0163] For example, the wearable device may further include at least one sensor. The instructions, when executed by the processor, may cause the wearable device to identify a location of the wearable device with respect to the zone based on sensor data obtained from the at least one sensor. The instructions, when executed by the processor, may cause the wearable device to execute a function for displaying one of a plurality of virtual objects, including the first virtual object and the second virtual object, associated with the object through the display, based on identifying that the location of the wearable device is included in the first zone among the plurality of zones included in the zone.

[0164] For example, the instructions, when executed by the processor, may cause the wearable device to display the first virtual object together with a camera image based on a field-of-view (FOV) of the wearable device corresponding to the first area.

[0165] For example, the instructions, when executed by the processor, may cause the wearable device to store, in the memory, information related to a plurality of virtual objects including the first virtual object and the second virtual object, received from an external electronic device. The instructions, when executed by the processor, may cause the wearable device to identify whether the first virtual object has been displayed on the display based on identifying the first area including the object through the camera. The instructions, when executed by the processor, may cause the wearable device to display the first screen including the first virtual object on the display based on identifying that the first virtual object has not been displayed on the display. The instructions, when executed by the processor, may cause the wearable device to display the third screen including the second virtual object through the display based on identifying that the first virtual object has been displayed through the display.

[0166] For example, the instructions, when executed by the processor, may cause the wearable device to transmit spatial information related to the real space to an external electronic device connected to the wearable device. The instructions, when executed by the processor, may cause the wearable device to display, through the display, a plurality of first virtual objects related to the real space, received from the external electronic device based on the spatial information. The instructions, when executed by the processor, may cause the wearable device to display, through the display, a plurality of second virtual objects related to one or more other wearable devices connected to the external electronic device, received through the external electronic device.

[0167] In one embodiment, a method of a wearable device may include an operation of defining a zone including a plurality of areas based on scanning of an actual space. The method may include an operation of displaying a first screen including a first virtual object related to an object in a first area, based on a first area identified through a camera of the wearable device among the plurality of areas, through a display of the wearable device. The operation may include an operation of, after displaying the first screen, displaying a second screen related to a second area, based on a second area identified through the camera among the plurality of areas, through the display. The operation may include an operation of, after displaying the second screen, displaying a third screen including a second virtual object related to the object, based on the first area identified through the camera, through the display.

[0168] For example, the operation of defining the zone including the plurality of areas may include an operation of identifying the plurality of areas using spatial information related to the actual space acquired through the camera.

[0169] For example, the operation of displaying the first screen through the display may include an operation of displaying the first screen through the display based on the first region identified through the first spatial information. The operation of displaying the second screen through the display may include an operation of displaying the second screen through the display based on the second region identified based on obtaining second spatial information having a similarity of less than or equal to a reference value with the first spatial information. The operation of displaying the third screen through the display may include an operation of displaying the third screen through the display based on the first region identified based on obtaining third spatial information having a similarity of more than or equal to a reference value with the first spatial information.

[0170] For example, the method may further include an operation of performing an event for determining, after displaying the second screen, whether to display the first virtual object related to the object within the first area through the display based on the first area identified through the camera. The method may further include an operation of displaying, in response to a user input for displaying the first virtual object, a fourth screen including the first virtual object through the display. The operation of displaying the third screen through the display may include an operation of displaying, in response to a user input for displaying the second virtual object, the third screen including the second virtual object through the display.

[0171] For example, the operation of displaying the first screen through the display may include an operation of identifying whether a time for which the first region is identified through the camera while the first screen including the first virtual object is displayed through the display is less than a reference time. The operation of displaying the third screen through the display may include an operation of displaying the third screen including the second virtual object through the display based on the first region identified after the second screen is displayed, based on identifying the time being less than the reference time. The method may further include an operation of displaying a fourth screen including the first virtual object through the display based on the first region identified after the second screen is displayed, based on identifying the time being greater than or equal to the reference time.

[0172] For example, the method may further include an operation of identifying, after displaying the second screen, whether all of the spatial information of the plurality of areas within the zone has been acquired based on spatial information related to the actual space acquired through the camera. The method may further include an operation of displaying, in response to identifying that all of the plurality of areas have been identified, at least one of the plurality of screens that were displayed through the display after the first screen was displayed and before the second screen was displayed. The operation of displaying the third screen through the display may include an operation of displaying, in response to identifying that not all of the spatial information has been acquired, the third screen including the second virtual object related to the object within the first area through the display.

[0173] For example, the method may further include an operation of identifying a location of the wearable device with respect to the zone based on sensor data acquired through at least one sensor of the wearable device. The method may further include an operation of executing a function for displaying one of a plurality of virtual objects including the first virtual object and the second virtual object related to the object through the display based on identifying that the location of the wearable device is included in the first zone among the plurality of zones included in the zone.

[0174] For example, the action of displaying the first screen through the display may include an action of displaying the first virtual object together with a camera image based on a field-of-view of the wearable device corresponding to the first area.

[0175] For example, the method may further include an operation of storing information related to a plurality of virtual objects including the first virtual object and the second virtual object, received from an external electronic device, in a memory of the wearable device. The method may further include an operation of identifying whether the first virtual object has been displayed through the display based on identifying the first area including the object through the camera. The operation of displaying the first screen through the display may include an operation of displaying the first screen including the first virtual object through the display based on identifying that the first virtual object has not been displayed through the display. The operation of displaying the third screen through the display may include an operation of displaying the third screen including the second virtual object through the display based on identifying that the first virtual object has been displayed through the display.

[0176] In one embodiment, a non-transitory computer readable storage medium storing one or more programs may include instructions that, when executed by a processor of a wearable device having a camera and a display, cause the wearable device to define an area including a plurality of areas based on scanning of a physical space. The one or more programs may include instructions that, when executed by the processor of the wearable device, cause the wearable device to display, through the display, a first screen including a first virtual object associated with an object within a first area identified through the camera among the plurality of areas. The one or more programs may include instructions that, when executed by the processor of the wearable device, cause the wearable device to display a second screen related to a second region identified through the camera among the plurality of regions after displaying the first screen. The one or more programs may include instructions that, when executed by the processor of the wearable device, cause the wearable device to display a third screen including a second virtual object related to the object through the display, based on the first region identified through the camera after displaying the second screen.

[0177] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0178] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0179] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0180] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0181] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In wearable devices, camera; display; Memory for storing instructions; and Contains a processor, The above instructions, when executed by the processor, cause the wearable device to: Define a region containing multiple areas based on scanning of the physical space, Based on a first region identified through the camera among the plurality of regions, a first screen including a first virtual object related to an object within the first region is displayed through the display, After displaying the first screen, based on a second area identified through the camera among the plurality of areas, a second screen related to the second area is displayed through the display, After displaying the second screen, a third screen including a second virtual object related to the object is displayed through the display based on the first area identified through the camera. Wearable devices.

2. In paragraph 1, The above instructions, when executed by the processor, cause the wearable device to: By using the spatial information related to the actual space acquired through the camera, causing the plurality of areas to be identified, Wearable devices.

3. In paragraph 2, The above instructions, when executed by the processor, cause the wearable device to: Based on the first area identified through the first spatial information, the first screen is displayed through the display, Based on the second area identified by obtaining second spatial information having a similarity less than a reference value with the first spatial information, the second screen is displayed through the display, Based on the first area identified by obtaining third spatial information having a similarity higher than a reference value with the first spatial information, causing the third screen to be displayed through the display, Wearable devices.

4. In any one of paragraphs 1 to 3, The above instructions, when executed by the processor, cause the wearable device to: After displaying the second screen, an event is performed to determine whether to display the first virtual object related to the object within the first area through the display based on the first area identified through the camera. In response to a user input for displaying the second virtual object, displaying the third screen including the second virtual object through the display; In response to a user input for displaying the first virtual object, causing a fourth screen including the first virtual object to be displayed through the display. Wearable devices.

5. In any one of paragraphs 1 to 4, The above instructions, when executed by the processor, cause the wearable device to: While the first screen including the first virtual object is displayed through the display, it is identified whether the time for which the first area is identified through the camera is less than a reference time, Based on identifying the time less than the reference time, the third screen including the second virtual object is displayed through the display based on the identified first area after the second screen is displayed, Based on identifying the time greater than or equal to the reference time, causing a fourth screen including the first virtual object to be displayed through the display based on the identified first area after the second screen is displayed. Wearable devices.

6. In any one of paragraphs 1 to 5, The above instructions, when executed by the processor, cause the wearable device to: After displaying the second screen, based on the spatial information related to the actual space acquired through the camera, it is identified whether all the spatial information of the plurality of areas within the zone has been acquired, In response to identifying that not all of the above spatial information has been acquired, displaying the third screen including the second virtual object related to the object within the first area through the display; In response to identifying that all of the above plurality of areas have been identified, causing at least one of the plurality of screens that were displayed through the display before the second screen was displayed after the first screen was displayed to be displayed. Wearable devices.

7. In any one of paragraphs 1 to 6, further comprising at least one sensor; The above instructions, when executed by the processor, cause the wearable device to: Identifying the location of the wearable device relative to the area based on sensor data acquired from at least one sensor; Based on identifying that the location of the wearable device is included in the first region among the plurality of regions included in the zone, causing a function to be executed to display one of the plurality of virtual objects including the first virtual object and the second virtual object related to the object through the display. Wearable devices.

8. In any one of paragraphs 1 to 7, The above instructions, when executed by the processor, cause the wearable device to: Causing to display the first virtual object together with the camera image based on the field-of-view (FOV) of the wearable device corresponding to the first area. Wearable devices.

9. In any one of paragraphs 1 to 8, The above instructions, when executed by the processor, cause the wearable device to: Store information related to a plurality of virtual objects, including the first virtual object and the second virtual object, received from an external electronic device in the memory, Based on identifying the first area including the object through the camera, identifying whether the first virtual object is displayed through the display, Based on identifying that the first virtual object is not displayed through the display, displaying the first screen including the first virtual object through the display, Based on identifying that the first virtual object has been displayed through the display, causing the third screen including the second virtual object to be displayed through the display. Wearable devices.

10. In any one of paragraphs 1 to 9, The above instructions, when executed by the processor, cause the wearable device to: Transmitting spatial information related to the actual space to an external electronic device connected to the wearable device; Displaying a plurality of first virtual objects related to the actual space, received from the external electronic device based on the above spatial information, through the display; Causing a plurality of second virtual objects related to one or more other wearable devices connected to the external electronic device, received through the external electronic device, to be displayed through the display. Wearable devices.

11. In the method of a wearable device, An action of defining a zone containing multiple areas based on scanning of a real space; An action of displaying a first screen including a first virtual object related to an object within a first region through a display of the wearable device based on a first region identified through a camera of the wearable device among the plurality of regions; After displaying the first screen, an operation of displaying a second screen related to the second area through the display based on a second area identified through the camera among the plurality of areas; and After displaying the second screen, an action is included to display a third screen including a second virtual object related to the object through the display based on the first area identified through the camera. method.

12. In the 11th paragraph, the operation of defining the zone including the plurality of areas is as follows: An operation of identifying the plurality of areas using spatial information related to the actual space acquired through the camera, method.

13. In the 12th paragraph, the operation of displaying the first screen through the display is as follows: An operation of displaying the first screen through the display based on the first area identified through the first spatial information is included. The action of displaying the above second screen through the above display is: An operation of displaying the second screen through the display based on the second area identified based on obtaining second spatial information having a similarity less than or equal to a reference value with the first spatial information is included. The action of displaying the above third screen through the above display is, An operation of displaying the third screen through the display based on the first area identified based on obtaining third spatial information having a similarity greater than a reference value with the first spatial information, method.

14. In any one of paragraphs 11 to 13, After displaying the second screen, an action of performing an event for determining whether to display the first virtual object related to the object within the first area through the display based on the first area identified through the camera; and In response to a user input for displaying the first virtual object, an operation of displaying a fourth screen including the first virtual object through the display; further comprising; The action of displaying the above third screen through the above display is, In response to a user input for displaying the second virtual object, an operation of displaying the third screen including the second virtual object through the display; method.

15. In any one of paragraphs 11 to 14, The action of displaying the above first screen through the above display is: An operation for identifying whether the time for which the first area is identified through the camera is less than a reference time while the first screen including the first virtual object is displayed through the display, The action of displaying the above third screen through the above display is, Based on identifying the time less than the reference time, including an action of displaying the third screen including the second virtual object through the display based on the identified first area after the second screen is displayed, The above method, Based on identifying the time greater than or equal to the reference time, further comprising an action of displaying a fourth screen including the first virtual object through the display based on the identified first area after the second screen is displayed. method.

Images