Wearable device for providing immersive experience and control method therefor

Abstract

Provided are a wearable device for providing an immersive experience and a control method therefor. The wearable device according to one embodiment of the present disclosure may: identify a real object located in the vicinity of a user on the basis of a real-world image acquired by at least one camera; identify the posture of the user wearing the wearable device on the basis of sensing data acquired by at least one sensor; determine a first immersion level on the basis of the real object and the posture of the user that have been identified; display an execution screen of at least one application as a virtual object on the basis of the determined first immersion level; determine a second immersion level on the basis of an interaction between the virtual object and the user while the virtual object is displayed; and change a display method of the virtual object on the basis of the determined second immersion level.

Description

Wearable device providing immersive experience and method for controlling the same

[0001] The present disclosure relates to a wearable device that provides an immersive experience and a method for controlling the same.

[0002] The variety of services and additional features offered through wearable devices, such as augmented reality (AR) and virtual reality (VR) devices, is steadily increasing. To enhance the utility of these devices and satisfy the diverse needs of users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices that offer diverse features and differentiate themselves from competitors. Consequently, the various functions offered through wearable devices are also becoming increasingly sophisticated.

[0003] The above information is provided solely as background information to assist in understanding the disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] Wearable devices can provide immersive views by relatively darkening or blurring the surroundings of running virtual objects (e.g., the running screen of an application). Conventional wearable devices can only switch to immersive views in certain situations, such as watching a movie. Consequently, users must manually change the viewing mode to enter an immersive viewing environment, which can be cumbersome.

[0005] The disclosure addresses at least the aforementioned problems and / or shortcomings and provides at least the advantages described below. Accordingly, according to one aspect of the present disclosure, a wearable device (e.g., augmented reality glasses configured to provide augmented reality and / or virtual reality environments) is provided, which is configured to provide an optimal viewing environment to a user based on the user's posture and the physical environment surrounding the user.

[0006] According to another aspect of the present disclosure, a wearable device is provided that is configured to provide an optimal viewing environment to a user by adaptively changing an immersive viewing environment based on interaction between a user and a virtual object (e.g., an application execution screen) displayed through the wearable device.

[0007] According to another aspect of the present disclosure, a method for controlling a wearable device (e.g., augmented reality glasses configured to provide an augmented reality and / or virtual reality environment) is provided, wherein the wearable device is configured to provide an optimal viewing environment to a user based on the posture of the user wearing the wearable device and the physical environment around the user.

[0008] According to one embodiment of the present disclosure, a control method of a wearable device may be provided that is set to provide an optimal viewing environment to a user by adaptively changing an immersive viewing environment based on an interaction between a user and a virtual object (e.g., an application execution screen) displayed through the wearable device.

[0009] Additional aspects will be partly explained in the following description, and partly will become apparent from the description or may be learned by practicing the embodiments presented.

[0010] According to one aspect of the present disclosure, a wearable device is provided. The wearable device includes at least one sensor, at least one camera, a memory storing at least one computer program, and at least one processor communicatively coupled to the at least one sensor, the at least one camera, and the memory, wherein the at least one computer program includes computer-executable instructions that, when individually or collectively executed by the at least one processor, cause the wearable device to identify a real-world object surrounding a user based on an image of the real world obtained through at least one camera, identify a posture of a user wearing the wearable device based on sensed data obtained by at least one sensor based on the identified real-world object and the posture of the user, determine a first immersion level based on the determined first immersion level, display an execution screen of at least one application as a virtual object while a virtual object is displayed, determine a second immersion level based on an interaction between the virtual object and the user, and change a display method of the virtual object based on the determined second immersion level.

[0011] According to another aspect of the present disclosure, a method performed by a wearable device is provided. The method includes: identifying, by the wearable device, a real-world object located around a user based on an image of the real world obtained through at least one camera of the wearable device; identifying, by the wearable device, a posture of a user wearing the wearable device based on sensed data obtained by at least one sensor of the wearable device, based on the identified real-world object and the posture of the user; determining, by the wearable device, a first immersion level based on the determined first immersion level; displaying, by the wearable device, an execution screen of at least one application as a virtual object while a virtual object is displayed; determining, by the wearable device, a second immersion level based on an interaction between the virtual object and the user; and changing, by the wearable device, a display method of the virtual object based on the determined second immersion level.

[0012] According to another aspect of the present disclosure, one or more non-transitory computer-readable storage media storing one or more computer programs include computer-executable instructions that, when individually or collectively executed by one or more processors of a wearable device, cause the wearable device to perform operations. The operations include an operation in which the wearable device identifies a real-world object located around a user based on an image of the real world obtained through at least one camera, an operation in which the wearable device identifies a posture of a user wearing the wearable device based on sensed data obtained through at least one sensor based on the identified real-world object and the posture of the user, an operation in which the wearable device determines a first immersion level based on a determined first immersion level, an operation in which the wearable device displays an execution screen of at least one application as a virtual object while a virtual object is displayed, an operation in which the wearable device determines a second immersion level based on an interaction between the virtual object and the user, and an operation in which the wearable device changes a display method of the virtual object based on the determined second immersion level.

[0013] According to one embodiment of the present disclosure, a wearable device is provided that provides a technical effect of providing an optimal viewing environment to a user based on the posture of the user wearing the wearable device (e.g., augmented reality glasses configured to provide an augmented reality and / or virtual reality environment) and the physical environment around the user.

[0014] According to one embodiment of the present disclosure, a wearable device adaptively changes an immersive viewing environment based on interaction between a user and a virtual object (e.g., an application execution screen) displayed through the wearable device, thereby providing a technical effect of providing an optimal viewing environment to the user.

[0015] Other aspects, advantages and important features of the present disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments of the disclosure taken in conjunction with the accompanying drawings.

[0016] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0017] FIG. 1 is a block diagram of an electronic device within a network environment according to an embodiment of the present disclosure.

[0018] FIG. 2A is a perspective view of a wearable device (e.g., an electronic device) according to one embodiment of the present disclosure.

[0019] FIG. 2b is a perspective view illustrating the internal configuration of a wearable device (e.g., an electronic device) according to one embodiment of the present disclosure.

[0020] FIG. 2c is an exploded perspective view of a wearable device (e.g., an electronic device) according to one embodiment of the present disclosure.

[0021] FIG. 3 illustrates an eye tracking camera structure of a wearable device according to one embodiment of the present disclosure.

[0022] FIG. 4 is a diagram for explaining a function or operation of a wearable device according to one embodiment of the present disclosure to provide a virtual object to a user based on a first immersion level and a second immersion level.

[0023] FIGS. 5A, 5B, and 5C are drawings for explaining, from a user interface perspective, a function or operation of a wearable device according to various embodiments of the present disclosure to display a virtual object (e.g., an execution screen of a gallery application) according to a first immersion level (e.g., level 4 as an initial level).

[0024] FIGS. 6A, 6B, and 6C are drawings for explaining, from a user interface perspective, a function or operation of a wearable device according to various embodiments of the present disclosure to display a virtual object (e.g., an execution screen of a gallery application) according to a first immersion level (e.g., level 6 as an initial level).

[0025] FIG. 7A is a diagram illustrating an immersive score for determining an immersion level (e.g., a first immersion level and / or a second immersion level) according to one embodiment of the present disclosure.

[0026] FIG. 7b is a diagram illustrating an immersion level (e.g., a first immersion level and / or a second immersion level) according to one embodiment of the present disclosure.

[0027] FIGS. 8A and 8B are drawings for explaining, from a user interface perspective, a function or operation of changing the display method of a virtual object based on an immersion level that changes as an immersion score increases when the type of application running on a wearable device according to various embodiments of the present disclosure is a first type application (e.g., an application for viewing).

[0028] FIGS. 9A, 9B, and 9C are drawings for explaining, from a user interface perspective, a function or operation of changing the display method of a virtual object based on an immersion level that changes as an immersion score increases when the type of application running on a wearable device according to various embodiments of the present disclosure is a second type application (e.g., a game application).

[0029] FIG. 10 is a drawing for explaining a function or operation in which a display method of a virtual object is changed according to an increase or reset of an immersion score when there are multiple virtual objects according to one embodiment of the present disclosure.

[0030] FIG. 11a, FIG. 11b, FIG. 11c, FIG. 11d, and FIG. 11e are drawings for explaining the function or operation described in FIG. 10 from a user interface perspective.

[0031] FIG. 12 is a diagram for explaining a function or operation in which a display method of a virtual object is maintained or changed according to an increase in an immersion score when an application running through a wearable device according to one embodiment of the present disclosure is a third type application (e.g., an application for initiation work).

[0032] FIG. 13a, FIG. 13b, FIG. 13c, and FIG. 13d are drawings for explaining the function or operation described in FIG. 12 from a user interface perspective.

[0033] FIG. 14 is a diagram illustrating a function or operation of a wearable device according to one embodiment of the present disclosure to change display properties of at least one virtual object and interface as the immersion level increases.

[0034] FIGS. 15A and 15B are drawings for explaining the functions or operations described in FIG. 14 according to various embodiments from a user interface perspective.

[0035] FIG. 16 is an exemplary drawing for explaining a function or operation of changing and displaying display properties of grouped and displayed virtual objects based on an interaction with at least one virtual object among grouped and displayed virtual objects, by a wearable device according to one embodiment of the present disclosure.

[0036] FIGS. 17A and 17B are drawings for explaining the functions or operations illustrated in FIG. 16 according to various embodiments from a user interface perspective.

[0037] FIG. 17c is a diagram illustrating a function or operation of a wearable device according to one embodiment of the present disclosure to display a virtual object in a full virtual reality based on a first immersion level exceeding a threshold level.

[0038] FIG. 18a and FIG. 18b are drawings for explaining another type of wearable device of the present disclosure,

[0039] FIG. 19 is a block diagram illustrating a wearable device according to one embodiment of the present disclosure.

[0040] The same reference numbers are used throughout the drawing to indicate the same elements.

[0041] The following description, with reference to the accompanying drawings, is provided to facilitate a comprehensive understanding of various embodiments of the present invention as defined by the claims and their equivalents. While it includes numerous specific details to aid understanding, these should be considered illustrative only. Accordingly, those skilled in the art will recognize that various changes and modifications to the various embodiments described herein may be made without departing from the scope and spirit of the present invention. Furthermore, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0042] The terms and words used in the following description and claims are not limited to their bibliographic meanings, but are merely used by the inventors to facilitate a clear and consistent understanding of the present invention. Therefore, those skilled in the art should understand that the following description of various embodiments of the present invention is provided for illustrative purposes only and is not intended to limit the present invention as defined by the appended claims and their equivalents.

[0043] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "component surfaces" includes reference to one or more of these surfaces.

[0044] It should be understood that each block in each flowchart and the combination of flowcharts can be performed by one or more computer programs containing instructions. The entirety of one or more computer programs may be stored in a single memory device, or the one or more computer programs may be divided into different parts stored in different memory devices.

[0045] The functions or tasks described herein may be performed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing and includes an application processor (AP, e.g., a central processing unit (CPU)), a communications processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, a Bluetooth® chip, a global positioning system (GPS) chip, a near-field communication (NFC) chip, a connection chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio codec chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on a chip (SoC), an IC, or similar circuits.

[0046] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment.

[0047] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0048] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0049] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0050] The memory (130) can store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., the program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a nonvolatile memory (134). The nonvolatile memory (134) can include an internal memory (136) and an external memory (138).

[0051] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0052] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0053] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0054] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0055] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0056] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0057] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0058] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0059] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0060] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0061] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0062] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0063] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a fifth-generation (5G) network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

[0064] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4th generation (4G) network, for example, new radio access technology (NR). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0065] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0066] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0067] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0068] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102 or 104) or the server (108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0069] FIG. 2A is a perspective view of a wearable device (200) according to one embodiment of the present disclosure.

[0070] Referring to FIG. 2A, the wearable device (200) is an electronic device in the form of glasses, which allows a user to visually perceive surrounding objects or the environment while wearing the wearable device (200). For example, the wearable device (200) may be a head-mounted device (HMD) or smart glasses that can directly provide images to the user's eyes. The configuration of the wearable device (200) of FIG. 2A may be all or part of the same as the configuration of the electronic device (101) of FIG. 1.

[0071] According to various embodiments, the wearable device (200) may include a housing (210) that forms the exterior of the wearable device (200). The housing (210) may provide a space in which components of the wearable device (200) may be placed. For example, the housing (210) may include a lens frame (202) and at least one wearing member (203).

[0072] According to various embodiments, the wearable device (200) may include a display member (201) capable of providing visual information to a user. For example, the display member (201) may include a module equipped with a lens, a display, a waveguide, and / or a touch circuit. According to one embodiment, the display member (201) may be formed transparently or translucently. According to one embodiment, the display member (201) may include a window member whose light transmittance can be adjusted by adjusting the concentration of a translucent glass material or a coloring material. According to one embodiment, the display members (201) may be provided as a pair, and may be arranged to correspond to the left and right eyes of the user, respectively, when the wearable device (200) is worn on the user's body.

[0073] According to various embodiments, the lens frame (202) can accommodate at least a portion of the indicator member (201). For example, the lens frame (202) can surround at least a portion of an edge of the indicator member (201). In one embodiment, the lens frame (202) can position at least one of the indicator members (201) to correspond to a user's eye. In one embodiment, the lens frame (202) can be a rim of a typical eyeglass structure. In one embodiment, the lens frame (202) can include at least one closed curve surrounding the indicator member (201).

[0074] According to various embodiments, the wearing member (203) may extend from the lens frame (202). For example, the wearing member (203) may extend from an end of the lens frame (202) and, together with the lens frame (202), may be supported or positioned on the user's body (e.g., an ear). According to one embodiment, the wearing member (203) may be rotatably coupled to the lens frame (202) via a hinge structure (229). According to one embodiment, the wearing member (203) may include an inner side (231c) configured to face the user's body and an outer side (231d) opposite the inner side.

[0075] According to various embodiments, the wearable device (200) may include a hinge structure (229) configured to fold the wearing member (203) relative to the lens frame (202). The hinge structure (229) may be positioned between the lens frame (202) and the wearing member (203). When the wearable device (200) is not being worn, the user may fold the wearing member (203) so that a portion overlaps the lens frame (202) and carry or store the device.

[0076] FIG. 2b is a perspective view illustrating the internal configuration of a wearable device according to one embodiment of the present disclosure. FIG. 2c is an exploded perspective view of a wearable device according to one embodiment of the present disclosure.

[0077] Referring to FIGS. 2B and 2C, the wearable device (200) may include components accommodated in a housing (210) (e.g., at least one circuit board (241) (e.g., a printed circuit board (PCB), a printed board assembly (PBA), a flexible PCB (FPCB), or a rigid-flexible PCB (RFPCB)), at least one battery (243), at least one speaker module (245), at least one power transmission structure (246), and / or a camera module (250)). The configuration of the housing (210) of FIG. 2B may be all or part of the same as the configuration of the display member (201), the lens frame (202), the wearing member (203), and the hinge structure (229) of FIG. 2A.

[0078] According to various embodiments, the wearable device (200) may acquire and / or recognize a visual image of an object or environment in a direction (e.g., -Y direction) that the user is looking at or that the wearable device (200) is facing by using a camera module (250) (e.g., the camera module (180) of FIG. 1), and may receive information about the object or environment from an external electronic device (e.g., the electronic device (102, 104) of FIG. 1 or the server (108)) through a network (e.g., the first network (198) or the second network (199) of FIG. 1). In one embodiment, the wearable device (200) may provide the received information about the object or environment to the user in an acoustic or visual form. The wearable device (200) may provide the received information about the object or environment to the user in a visual form through a display member (201) by using a display module (e.g., the display module (160) of FIG. 1). For example, the wearable device (200) can implement augmented reality by visualizing information about objects or the environment and combining it with actual images of the user's surroundings.

[0079] According to various embodiments, the display member (201) may include a first side (F1) facing a direction in which external light is incident (e.g., -Y direction) and a second side (F2) facing a direction opposite to the first side (F1) (e.g., +Y direction). When a user wears the wearable device (200), at least a portion of light or an image incident through the first side (F1) may pass through the second side (F2) of the display member (201) arranged to face the user's left eye and / or right eye and be incident on the user's left eye and / or right eye.

[0080] According to various embodiments, the lens frame (202) may include at least two frames. For example, the lens frame (202) may include a first frame (202a) and a second frame (202b). According to one embodiment, when a user wears the wearable device (200), the first frame (202a) may be a frame that faces the user's face, and the second frame (202b) may be a part of the lens frame (202) that is spaced apart from the first frame (202a) in a direction of the user's gaze (e.g., -Y direction).

[0081] According to various embodiments, the light output module (211) can provide images and / or videos to the user. For example, the light output module (211) can include a display panel (not shown) capable of outputting videos, and a lens (not shown) corresponding to the user's eyes and guiding the videos to the display member (201). For example, the user can obtain videos output from the display panel of the light output module (211) through the lens of the light output module (211). According to various embodiments, the light output module (211) can include a device configured to display various pieces of information. For example, the light output module (211) can include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). According to one embodiment, when the light output module (211) and / or the display member (201) includes one of an LCD, a DMD, or an LCoS, the wearable device (200) may include a light source that irradiates light to a display area of ​​the light output module (211) and / or the display member (201). According to one embodiment, when the light output module (211) and / or the display member (201) includes one of an OLED or a micro LED, the wearable device (200) may provide a virtual image to a user without including a separate light source.

[0082] According to various embodiments, at least a portion of the light output module (211) may be disposed within the housing (210). For example, the light output module (211) may be disposed on the wearable member (203) or the lens frame (202) to correspond to the user's right eye and left eye, respectively. According to one embodiment, the light output module (211) may be connected to the display member (201) and provide an image to the user through the display member (201).

[0083] According to various embodiments, the circuit board (241) may include components for driving the wearable device (200). For example, the circuit board (241) may include at least one integrated circuit chip, and at least one of the processor (120), the memory (130), the power management module (188), or the communication module (190) of FIG. 1 may be provided to the integrated circuit chip. According to one embodiment, the circuit board (241) may be disposed within the wearing member (203) of the housing (210). According to one embodiment, the circuit board (241) may be electrically connected to the battery (243) through the power transmission structure (246). According to one embodiment, the circuit board (241) is connected to a flexible printed circuit board (205) and can transmit electrical signals to electronic components of the electronic device (e.g., a light output module (211), a camera module (250), and a light emitting unit) through the flexible printed circuit board (205). According to one embodiment, the circuit board (241) may be a circuit board including an interposer.

[0084] According to various embodiments, the flexible printed circuit board (205) may extend from the circuit board (241) across the hinge structure (229) into the interior of the lens frame (202) and may be disposed around at least a portion of the perimeter of the indicator member (201) within the lens frame (202).

[0085] According to various embodiments, the battery (243) (e.g., battery (189) of FIG. 1) may be electrically connected to components of the wearable device (200) (e.g., light output module (211), circuit board (241), speaker module (245), microphone module (247), and / or camera module (250)) and may supply power to the components of the wearable device (200).

[0086] According to various embodiments, at least a portion of the battery (243) may be disposed on the wearable member (203). In one embodiment, the battery (243) may be disposed on an end (203a, 203b) of the wearable member (203). For example, the battery (243) may include a first battery (243a) disposed on a first end (203a) of the wearable member (203) and a second battery (243b) disposed on a second end (203b).

[0087] According to various embodiments, the speaker module (245) (e.g., the audio module (170) or the sound output module (155) of FIG. 1) may convert an electrical signal into sound. At least a portion of the speaker module (245) may be disposed within the wearable member (203) of the housing (210). In one embodiment, the speaker module (245) may be positioned within the wearable member (203) so as to correspond to the user's ear. For example, the speaker module (245) may be disposed between the circuit board (241) and the battery (243).

[0088] According to various embodiments, the power transmission structure (246) can transmit power from the battery (243) to an electronic component (e.g., an optical output module (211)) of the wearable device (200). For example, the power transmission structure (246) is electrically connected to the battery (243) and / or the circuit board (241), and the circuit board (241) can transmit power received through the power transmission structure (246) to the optical output module (211). According to one embodiment, the power transmission structure (246) can be connected to the circuit board (241) through the speaker module (245). For example, when the wearable device (200) is viewed from the side (e.g., in the Z-axis direction), the power transmission structure (246) can at least partially overlap the speaker module (245).

[0089] According to various embodiments, the power transmission structure (246) may be a configuration capable of transmitting power. For example, the power transmission structure (246) may include a flexible printed circuit board or wires. For example, the wires may include a plurality of cables (not shown). In various embodiments, the shape of the power transmission structure (246) may be varied in various ways, taking into account the number and / or type of cables.

[0090] According to various embodiments, the microphone module (247) (e.g., the input module (150) and / or the audio module (170) of FIG. 1) may convert sound into an electrical signal. According to one embodiment, the microphone module (247) may be disposed on at least a portion of the lens frame (202). For example, at least one microphone module (247) may be disposed on the bottom (e.g., in the direction toward the -X axis) and / or the top (e.g., in the direction toward the X axis) of the wearable device (200). According to various embodiments, the wearable device (200) may recognize the user's voice more clearly by using voice information (e.g., sound) acquired from the at least one microphone module (247). For example, the wearable device (200) may distinguish voice information from ambient noise based on the acquired voice information and / or additional information (e.g., low-frequency vibration of the user's skin and bones). For example, the wearable device (200) can clearly recognize the user's voice and perform a function of reducing ambient noise (e.g., noise cancellation). The microphone module (247) according to various embodiments of the present disclosure may include a plurality of microphone modules (247) to perform beamforming. The microphone module (247) according to various embodiments of the present disclosure may include an omnidirectional or directional microphone.

[0091] According to various embodiments, the camera module (250) can capture still images and / or moving images. The camera module (250) may include at least one of a lens, at least one image sensor, an image signal processor, or a flash. According to one embodiment, the camera module (250) may be disposed within a lens frame (202) and may be disposed around the display member (201).

[0092] According to various embodiments, the camera module (250) may include at least one first camera module (251). In one embodiment, the first camera module (251) may capture a trajectory of a user's eye (e.g., pupil) or gaze. For example, the first camera module (251) may capture a reflection pattern of light emitted by a light emitting unit toward the user's eye. For example, the light emitting unit may emit light in an infrared band for tracking a trajectory of gaze using the first camera module (251). For example, the light emitting unit may include an IR LED. In one embodiment, a processor (e.g., processor (120) of FIG. 1) may adjust the position of the virtual image projected on the display member (201) so that the virtual image corresponds to a direction in which the user's pupil is looking. According to one embodiment, the first camera module (251) may include a global shutter (GS) type camera, and may track the trajectory of the user's eyes or gaze using a plurality of first camera modules (251) having the same specifications and performance.

[0093] According to various embodiments, the first camera module (251) may periodically or aperiodically transmit information related to the trajectory of the user's eyes or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1). According to one embodiment, the first camera module (251) may transmit the trajectory information to the processor when it detects that the user's gaze has changed (e.g., the eyes move more than a reference value while the head is still) based on the trajectory information.

[0094] According to various embodiments, the camera module (250) may include a second camera module (253). According to one embodiment, the second camera module (253) may capture external images. According to one embodiment, the second camera module (253) may be a camera of a global shutter type or a rolling shutter (RS) type. According to one embodiment, the second camera module (253) may capture external images through a second optical hole (223) formed in the second frame (202b). For example, the second camera module (253) may include a high-resolution color camera and may be a high-resolution (HR) or photo video (PV) camera. In addition, the second camera module (253) may provide an auto focus (AF) function and an optical image stabilizer (OIS) function.

[0095] According to various embodiments, the wearable device (200) may include a flash (not shown) positioned adjacent to the second camera module (253). For example, the flash (not shown) may provide light to increase the brightness (e.g., illuminance) around the wearable device (200) when the second camera module (253) acquires an external image, and may reduce difficulties in acquiring images due to dark environments, mixing of various light sources, and / or reflection of light.

[0096] According to various embodiments, the camera module (250) may include at least one third camera module (255). According to one embodiment, the third camera module (255) may capture a user's action through the first optical hole (221) formed in the lens frame (202). For example, the third camera module (255) may capture a user's gesture (e.g., hand motion). The third camera module (255) and / or the first optical hole (221) may be respectively disposed at opposite side ends of the lens frame (202) (e.g., the second frame (202b)), for example, at opposite ends of the lens frame (202) (e.g., the second frame (202b)) in the X direction. According to one embodiment, the third camera module (255) may be a global shutter (GS) type camera. For example, the third camera module (255) can provide 360-degree space (e.g., omnidirectional), position recognition and / or movement recognition with a camera that supports 3DoF (degrees of freedom) or 6DoF. According to one embodiment, the third camera module (255) can perform a movement path tracking function (simultaneous localization and mapping, SLAM) and a user movement recognition function using a plurality of global shutter type cameras of the same standard and performance as a stereo camera. According to one embodiment, the third camera module (255) can include an infrared (IR) camera (e.g., a time of flight (TOF) camera or a structured light camera). For example, the IR camera can operate as at least a part of a sensor module (e.g., the sensor module (176) of FIG. 1) for detecting a distance to a subject.

[0097] According to one embodiment, at least one of the first camera module (251) or the third camera module (255) may be replaced with a sensor module (e.g., the sensor module (176) of FIG. 1) (e.g., a light detection and ranging (LiDAR) sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode. For example, the photodiode may include a positive intrinsic negative (PIN) photodiode or an avalanche photodiode (APD). The photodiode may be referred to as a photo detector or a photo sensor.

[0098] According to one embodiment, at least one of the first camera module (251), the second camera module (253), or the third camera module (255) may include a plurality of camera modules (not shown). For example, the second camera module (253) may be configured with a plurality of lenses (e.g., wide-angle and telephoto lenses) and image sensors and may be arranged on one side (e.g., the side facing the -Y axis) of the wearable device (200). For example, the wearable device (200) may include a plurality of camera modules, each having a different property (e.g., angle of view) or function, and may be controlled to change the angle of view of the camera module based on a user's selection and / or trajectory information. For example, at least one of the plurality of camera modules may be a wide-angle camera, and at least another may be a telephoto camera.

[0099] According to various embodiments, a processor (e.g., processor (120) of FIG. 1) may determine movement of the wearable device (200) and / or movement of the user by using information of the wearable device (200) acquired using at least one of a gesture sensor, a gyro sensor, or an acceleration sensor of a sensor module (e.g., sensor module (176) of FIG. 1) and a user's motion (e.g., approach of the user's body to the wearable device (200)) acquired using a second camera module (253). According to one embodiment, the wearable device (200) may include, in addition to the described sensors, a magnetic (geomagnetic) sensor capable of measuring direction using a magnetic field and magnetic lines, and / or a Hall sensor capable of acquiring movement information (e.g., direction of movement or distance of movement) using the strength of a magnetic field. For example, the processor can determine movement of the wearable device (200) and / or movement of the user based on information obtained from a magnetic (geomagnetic) sensor and / or a Hall sensor.

[0100] According to various embodiments (not shown), the wearable device (200) can perform an input function (e.g., a touch and / or pressure sensing function) that enables interaction with a user. For example, a component configured to perform a touch and / or pressure sensing function (e.g., a touch sensor and / or a pressure sensor) may be disposed on at least a portion of the wearable member (203). The wearable device (200) can control a virtual image output through the display member (201) based on information acquired through the component. For example, the sensor related to the touch and / or pressure sensing function may be configured in various ways, such as a resistive type, a capacitive type, an electromagnetic induction (EM) type, or an optical type. According to one embodiment, the component configured to perform the touch and / or pressure sensing function may be all or partly identical to the configuration of the input module (150) of FIG. 1.

[0101] According to various embodiments, the wearable device (200) may include a reinforcing member (260) disposed in the internal space of the lens frame (202) and formed to have a rigidity higher than the rigidity of the lens frame (202).

[0102] According to various embodiments, the wearable device (200) may include a lens structure (270). The lens structure (270) may refract at least a portion of light. For example, the lens structure (270) may be a prescription lens having refractive power. According to one embodiment, the lens structure (270) may be positioned behind (e.g., in the +Y direction) the second window member of the display member (201). For example, the lens structure (270) may be positioned between the display member (201) and the user's eye. For example, the lens structure (270) may face the display member.

[0103] According to various embodiments, the housing (210) may include a hinge cover (227) that may conceal a portion of the hinge structure (229). Another portion of the hinge structure (229) may be accommodated or concealed between the inner case (231) and the outer case (233), which will be described later.

[0104] According to various embodiments, the wearable member (203) may include an inner case (231) and an outer case (233). The inner case (231) is, for example, a case configured to face or directly contact the user's body, and may be made of a material with low thermal conductivity, for example, a synthetic resin. According to one embodiment, the inner case (231) may include an inner side facing the user's body (e.g., the inner side (231c) of FIG. 2A). The outer case (233) may include, for example, a material capable of at least partially transmitting heat (e.g., a metal material) and may be coupled to face the inner case (231). According to one embodiment, the outer case (233) may include an outer side opposite the inner side (231c) (e.g., the outer side (231d) of FIG. 2A). In one embodiment, at least one of the circuit board (241) or the speaker module (245) may be accommodated in a space separated from the battery (243) within the wearable member (203). In the illustrated embodiment, the inner case (231) may include a first case (231a) including the circuit board (241) or the speaker module (245) and a second case (231b) accommodating the battery (243), and the outer case (233) may include a third case (233a) coupled to face the first case (231a) and a fourth case (233b) coupled to face the second case (231b). For example, a first case (231a) and a third case (233a) may be combined (hereinafter, “first case portion (231a, 233a)”) to accommodate a circuit board (241) and / or a speaker module (245), and a second case (231b) and a fourth case (233b) may be combined (hereinafter, “second case portion (231b, 233b)”) to accommodate a battery (243).

[0105] According to various embodiments, the first case portion (231a, 233a) is rotatably connected to the lens frame (202) via a hinge structure (229), and the second case portion (231b, 233b) can be connected or mounted to an end of the first case portion (231a, 233a) via a connecting member (235). In some embodiments, among the connecting members (235), a portion that comes into contact with the user's body can be made of a material having low thermal conductivity, for example, an elastic material such as silicone, polyurethane, or rubber, and a portion that does not come into contact with the user's body can be made of a material having high thermal conductivity, for example, a metal material. For example, when heat is generated in the circuit board (241) or the battery (243), the connecting member (235) can block the heat from being transferred to the portion that comes into contact with the user's body, and can disperse or release the heat through the portion that does not come into contact with the user's body. According to one embodiment, a portion of the connecting member (235) that is configured to come into contact with the user's body may be interpreted as a part of the inner case (231), and a portion of the connecting member (235) that is not configured to come into contact with the user's body may be interpreted as a part of the outer case (233). According to one embodiment (not shown), the first case (231a) and the second case (231b) may be configured as an integral body without the connecting member (235), and the third case (233a) and the fourth case (233b) may be configured as an integral body without the connecting member (235). According to various embodiments, in addition to the illustrated components, other components (e.g., the antenna module (197) of FIG. 1) may be further included, and information about objects or environments may be provided from an external electronic device (e.g., the electronic device (102, 104) or server (108) of FIG. 1) through a network (e.g., the first network (198) or the second network (199) of FIG. 1) using the communication module (190).

[0106] In FIGS. 2A to 2C, only the wearable device (200) is illustrated and described, but it is not limited thereto, and some components of the wearable device (200) illustrated in FIGS. 2A to 2C may also be included in electronic devices such as smartphones and tablet PCs.

[0107] According to various embodiments of the present disclosure, a wearable device (200) can identify whether a user (210) is wearing the wearable device (200) through a proximity sensor included in the wearable device (200). Alternatively, according to various embodiments of the present disclosure, a wearable device (200) can determine whether the wearable device (200) is being worn by a user based on whether the frame of the wearable device (200) is unfolded (e.g., unfolded state) and whether proximity of the user is detected while the frame of the wearable device (200) is unfolded, through an angle sensor provided in a hinge portion of the wearable device (200).

[0108] FIG. 3 illustrates an eye tracking camera structure of a wearable device according to one embodiment.

[0109] Referring to FIG. 3, a wearable device (300) (e.g., a glasses-type device) may include an ET (eye tracking) camera (310), a display (321), an input optical member (322), a first waveguide (323), an output optical member (324), a first splitter (341), a second waveguide (342), and a second splitter (343).

[0110] According to various embodiments, the user's pupil (330) may be captured by the ET camera (310) through a first splitter (341) (e.g., a splitter for gaze tracking), a second waveguide (342), and a second splitter (343). The ET camera (310) may detect the pupil (330) in the captured image and track the user's gaze by confirming the movement of the detected pupil (330).

[0111] According to various embodiments, an image output through the display (321) may be reflected through the input optical member (322) and the first waveguide (323) and displayed through the output optical member (324). The wearable device (300) may output an image through the display (321) and at the same time track the user's gaze by checking the movement of the user's pupils (330).

[0112] FIG. 4 is a drawing for explaining a function or operation of a wearable device (200) according to one embodiment of the present disclosure that provides a virtual object (e.g., a first application execution screen (530)) to a user based on an immersion level (e.g., a first immersion level and a second immersion level).

[0113] FIGS. 5A to 5C are drawings for explaining, from a user interface perspective, a function or operation of a wearable device according to various embodiments of the present disclosure to display a virtual object (e.g., an execution screen of a gallery application) according to a first immersion level (e.g., level 4 as an initial level).

[0114] FIGS. 6A, 6B, and 6C are exemplary drawings for explaining, from a user interface perspective, a function or operation of a wearable device according to various embodiments of the present disclosure to display a virtual object (e.g., an execution screen of a gallery application) according to a first immersion level (e.g., level 6 as an initial level).

[0115] According to the immersion score or immersion level mentioned in the present disclosure, the display properties of a virtual object (e.g., the execution screen of a gallery application) may be changed, or the surroundings of the virtual object (e.g., the execution screen of a gallery application) may be changed to a virtual reality. The term immersion level mentioned in the present disclosure may be an immersion environment determination model used to change the properties of a virtual object (e.g., the first application execution screen (530)) and / or the surroundings of a virtual object (e.g., the first application execution screen (530)) that are shown to the user in a specified manner (e.g., a pass-through manner). For example, as the immersion level according to one embodiment of the present disclosure increases, the properties (e.g., the resolution and / or the size) of a virtual object (e.g., the first application execution screen (530)) that was shown to the user in a pass-through manner may be changed, or at least a part of the environment around the virtual object (e.g., the first application execution screen (530)) may be changed to a virtual reality environment. The immersion level according to one embodiment of the present disclosure may be determined by the immersion level determination logic (1960). Additionally, the function or operation of the wearable device (200) displaying a virtual object based on the immersion level according to one embodiment of the present disclosure may be performed by the immersion level reflection logic (1970).

[0116] Referring to FIGS. 4, 5A, 5B, and 5C, and 6A, 6B, and 6C, a wearable device (200) according to an embodiment of the present disclosure may, in operation 410, identify at least one real object (500) located around a user wearing the wearable device (200) based on an image of the real world acquired through at least one camera (e.g., a second camera module (253)). The wearable device (200) according to an embodiment of the present disclosure (e.g., the processor (120) of FIG. 1) may identify at least one real object (500) (e.g., a sofa (501), a light (502), a picture frame (503), a flower pot (504)) located around the user, thereby identifying a current location (e.g., a living room) where a user wearing the wearable device (200) is located. A wearable device (200) according to one embodiment of the present disclosure can obtain or infer information about a location where a user is currently located based on at least one identified real object (500) using a pre-stored database and / or generative artificial intelligence. For example, a wearable device (200) according to one embodiment of the present disclosure can identify or infer that a user wearing the wearable device (200) is currently located in a living room by identifying a sofa (501), a lamp (502), a picture frame (503), and a flower pot (504). A wearable device (200) according to one embodiment of the present disclosure (e.g., a processor (120) of FIG. 1 ) can identify a current location (e.g., a bedroom) where a user wearing the wearable device (200) is currently located by identifying at least one real object (500) (e.g., a bed (505), a pillow (506)) located around the user. For example, a wearable device (200) according to one embodiment of the present disclosure can identify or infer that a user wearing the wearable device (200) is currently located in a bedroom by identifying a bed (505) and a pillow (506).A wearable device (200) according to one embodiment of the present disclosure may determine the location where the user is currently located (e.g., through communication with a real object around the user) using short-range communication (e.g., NFC, WiFi (wireless fidelity) direct). The function or operation of the wearable device (200) according to one embodiment of the present disclosure to recognize a real object may be performed by the object recognition logic (1942). In addition, the function or operation of recognizing the location of the wearable device (200) may be performed by the location information recognition logic (1944).

[0117] According to one embodiment of the present disclosure, a wearable device (200) can identify a posture of a user wearing the wearable device (200) based on sensing data acquired by at least one sensor (e.g., the sensor module (176) of FIG. 1 and / or at least one camera) in operation 420. The function or operation of identifying the posture of the user according to one embodiment of the present disclosure can be performed by, for example, a posture recognizer (1932). According to one embodiment of the present disclosure, a wearable device (200) can identify or infer the current posture of the user by using information such as a look-up table in which a relationship between the sensing data and the posture of the user is defined. For example, the wearable device (200) according to one embodiment of the present disclosure can identify that the user is currently sitting on the sofa (501) by using images acquired by at least one camera (e.g., the second camera module (253)) and sensing data acquired by an acceleration sensor and / or a gyro sensor. Referring to FIG. 5B, a state in which a user wearing the wearable device (200) is sitting on the sofa (501) is exemplarily illustrated. The wearable device (200) according to one embodiment of the present disclosure can display a home screen including at least one application shortcut icon (e.g., the first icon (510)) and a tray (520) displaying at least one application icon as a virtual object on the real world after the performance of operations 410 and 420 is completed. Alternatively, the wearable device (200) according to one embodiment of the present disclosure may display a home screen including at least one application shortcut icon (e.g., the first icon (510)) and a tray (520) displaying at least one application icon as a virtual object in the real world after the user's wearing of the wearable device (200) is detected, regardless of the performance of operations 410 and 420. In FIG. 5B, an embodiment in which the home screen is displayed after the user sits down is exemplarily illustrated.

[0118] Referring to FIG. 6B, a wearable device (200) according to an embodiment of the present disclosure can identify or infer that a user is currently lying on a bed (506) by using images acquired by at least one camera (e.g., a second camera module (253)) and sensing data acquired by an acceleration sensor and / or a gyro sensor. After identifying the user's posture, the wearable device (200) according to an embodiment of the present disclosure can display a home screen including at least one application shortcut icon (e.g., a first icon (510)) and a tray (520) displaying at least one application icon as a virtual object on the real world.

[0119] According to one embodiment of the present disclosure, a wearable device (200) may determine a first immersion level (e.g., an initial level) based on a real object identified according to operation 410 and / or a posture of a user identified according to operation 420, in operation 430. However, the function or operation of determining the first immersion level based on the real object identified according to operation 410 and / or the posture of a user identified according to operation 420 is exemplary for describing various embodiments of the present disclosure. According to one embodiment of the present disclosure, the first immersion level may also be determined based on the type of an application being executed. For example, the wearable device (200) according to one embodiment of the present disclosure may display a virtual object (e.g., an execution screen of a gallery application) on the real world (e.g., a first immersion level is set to 1) when the type of the running application is an immersive application (e.g., a gallery application), and may display a virtual object (e.g., an execution screen of a game application) on a complete virtual reality (e.g., an immersion level is set to 10) when the type of the running application is an experiential application (e.g., a game application). The wearable device (200) according to one embodiment of the present disclosure may, in operation 440, display an execution screen of at least one application (e.g., an application execution screen (530)) as a virtual object based on the determined first immersion level. The wearable device (200) according to one embodiment of the present disclosure may calculate an immersion score based on a current location and / or posture of a user wearing the wearable device (200). A wearable device (200) according to one embodiment of the present disclosure can determine an immersion score based on information such as the table shown in FIG. 7a.

[0120] FIG. 7A is a diagram illustrating an immersive score for determining an immersion level (e.g., a first immersion level and / or a second immersion level) according to one embodiment of the present disclosure.

[0121] Referring to FIG. 7A, a wearable device (200) according to an embodiment of the present disclosure may determine an immersion score as 4 points using the information shown in FIG. 7A and the mathematical expression 1 below when a user is seated on a sofa (501) in a resting environment (e.g., a living room). As shown in FIG. 7A, if the immersion score determination criteria are determined differently depending on the type of application (e.g., whether it is an appreciation-type application such as a gallery application or an experiential application such as a game application), the wearable device (200) according to an embodiment of the present disclosure may calculate an immersion score for determining an immersion level (e.g., a first immersion level and / or a second immersion level) based on any one of the specified criteria (e.g., an immersion score calculation criterion for an appreciation-type application).

[0122]

[0123] In mathematical expression 1, can mean a score function for the user's posture, and can include a function in which the score is high when the state is static and low when the state is dynamic. In mathematical expression 1, may refer to a score function for the user's surrounding environment, and may include a function in which the score is determined based on the characteristics of the location. In mathematical expression 1, may refer to an interrupt score function, and may include a function set to lower the immersion level when a call requiring a user response occurs. In mathematical expression 1, a, b, and c may refer to weights between each element, and for example, a+b+c may be 1.

[0124] A wearable device (200) according to one embodiment of the present disclosure can determine an immersion level based on a calculated immersion score.

[0125] FIG. 7b is a diagram illustrating an immersion level (e.g., a first immersion level and / or a second immersion level) according to one embodiment of the present disclosure.

[0126] The immersion score and immersion level according to one embodiment of the present disclosure may be associated with each other and stored in the wearable device (200) and / or an external device. For example, the association may be such that when the immersion score is 1, it may be designated as immersion level 1, and when the immersion score is 2, it may be designated as immersion level 2. The function or operation of recognizing an external device according to one embodiment of the present disclosure may be performed by the external device recognition logic (1946). Referring to FIG. 7B, the immersion level has been described on the premise that multiple virtual objects (e.g., execution screens of multiple applications) are displayed as virtual objects, but the same may be applied to a case where a single application execution screen is displayed as a virtual object. The wearable device (200) according to one embodiment of the present disclosure may determine the first immersion level as level 4 when the immersion score is calculated as 4. According to one embodiment of the present disclosure, a wearable device (200) may display a virtual object according to a virtual object display environment corresponding to level 4, as shown in FIG. 5C. Level 4 according to one embodiment of the present disclosure may include, for example, a level in which a virtual object (e.g., an application execution screen) is displayed in a size enlarged by a specified ratio from an initially displayed size, and a tray (520) is displayed in a blurry or darker state than the initially set brightness or clarity. Referring to FIG. 5C, an embodiment is exemplarily illustrated in which, based on the virtual object being displayed according to level 4, the tray (520) is displayed in a blurry or darker state than the initially set brightness or clarity, and a first application execution screen (530) is displayed in a larger state by a specified ratio than the initially set size while maintaining the resolution.According to an embodiment of the present disclosure, a wearable device (200) may determine the size of a virtual object (e.g., how large to display it) and / or the environment around the virtual object (e.g., whether to change it to virtual reality) based on an immersion level corresponding to an immersion score to display the virtual object. Referring to FIG. 6C, when the immersion level (e.g., first immersion level) is set to 6, the wearable device (200) according to an embodiment of the present disclosure may display the size of a virtual object (e.g., an application execution screen) enlarged by a specified ratio from the initially displayed size, while changing a first area having a specified area around the virtual object into a virtual reality including a specified image and displaying it. In addition, the wearable device (200) according to an embodiment of the present disclosure may control the display module (160) so that the tray (520) is displayed more dimly or darker than the initially set brightness or clarity.

[0127] According to one embodiment of the present disclosure, the wearable device (200) may determine a second immersion level based on the interaction between the virtual object and the user while the virtual object is displayed, in operation 450. According to one embodiment of the present disclosure, the wearable device (200) may change the display method of the virtual object being displayed according to the first immersion level based on the determined second immersion level, in operation 460. According to one embodiment of the present disclosure, the wearable device (200) may determine the type of the currently running application (e.g., by the application characteristic determination logic (1952)) to determine the second immersion level. For example, the wearable device (200) according to one embodiment of the present disclosure may determine whether the currently running application is an appreciation-type application or an experiential application.

[0128] FIGS. 8A and 8B are drawings for explaining, from a user interface perspective, a function or operation of changing the display method of a virtual object based on an immersion level that changes as an immersion score increases when the type of application running on a wearable device (200) according to various embodiments of the present disclosure is a first type application (e.g., an application for viewing).

[0129] Referring to FIG. 8A, a wearable device (200) according to an embodiment of the present disclosure can display virtual objects and interfaces according to a first immersion level of level 4. A virtual object (e.g., a first application execution screen (530)) according to an embodiment of the present disclosure can be displayed in a pass-through manner while a user is wearing the wearable device (200). In other words, a virtual object (e.g., a first application execution screen (530)) and the real world can be simultaneously shown to the user. As the immersion level mentioned in the present disclosure increases (e.g., when the immersion level increases from level 4 (e.g., a first immersion level) to level 6 (e.g., a second immersion level)), the surroundings of a virtual object (e.g., a first application execution screen (530)) shown to the user in a pass-through manner can be gradually and automatically (e.g., without a designated input from the user to the wearable device (200)) changed into a virtual reality environment. According to one embodiment of the present disclosure, a virtual object (e.g., a first application execution screen (530)) may also move together (e.g., be displayed in a body-locked manner) as the user's gaze moves. In this case, a virtual reality environment provided around the virtual object (e.g., the first application execution screen (530)) may also move together and be provided to the user as the virtual object (e.g., the first application execution screen (530)) moves. Alternatively, according to one embodiment of the present disclosure, the display position of the virtual object (e.g., the first application execution screen (530)) may be displayed in a fixed manner (e.g., be displayed in a world-locked manner) as the user's gaze moves, and only the virtual reality environment may be gradually provided as the user's gaze moves. The user's gaze according to one embodiment of the present disclosure may be tracked, for example, by a gaze tracker (1934).

[0130] According to an embodiment of the present disclosure, when the wearable device (200) identifies that the user's gaze is maintained on the first application execution screen (530) for a specified period of time or longer, the wearable device (200) may increase the immersion score by 1 point for each specified period of time. Accordingly, the wearable device (200) according to an embodiment of the present disclosure may display a virtual object in an immersive view state, as illustrated in FIG. 8B, for example, when the immersion score increases from 4 points (e.g., the immersion score corresponding to the first immersion level) to 6 points. For example, the wearable device (200) according to an embodiment of the present disclosure may enlarge the first application execution screen (530) by a specified ratio or more from an initially set size (e.g., the size of the screen displayed when the immersion level is 1), and display the first application execution screen (530) by relatively darkening or blurring the area around the first application execution screen (530) (e.g., by applying a first visual effect (540).

[0131] FIGS. 9A, 9B, and 9C are drawings for explaining, from a user interface perspective, a function or operation of changing the display method of a virtual object based on an immersion level that changes as an immersion score increases when the type of application running on a wearable device (200) according to various embodiments of the present disclosure is a second type application (e.g., a game application).

[0132] Referring to FIG. 9A, a wearable device (200) according to an embodiment of the present disclosure can display virtual objects and interfaces according to a first immersion level of level 4. When a user continuously inputs input to control an application, the wearable device (200) according to an embodiment of the present disclosure can increase the immersion score by 1 point from the score corresponding to level 4 at a specified time interval. Accordingly, the wearable device (200) according to an embodiment of the present disclosure can display virtual objects in an immersive view state, as illustrated in FIG. 9B, when the immersion score becomes 6 points, for example. For example, the wearable device (200) according to one embodiment of the present disclosure may enlarge the first application execution screen (530) (e.g., active object) by a specified ratio or more than the initially set size (e.g., the size of the screen displayed when the immersion level is 1) and display the first application execution screen (530) by relatively darkening or blurring the area around the first application execution screen (530) (e.g., by applying the first visual effect (540) or by switching to a virtual reality environment). The wearable device (200) according to one embodiment of the present disclosure may increase the immersion score by 1 point at specified intervals when a user continuously inputs an input for controlling an application in an immersive view state as illustrated in FIG. 9b. Accordingly, the wearable device (200) according to one embodiment of the present disclosure may, for example, display a virtual object larger in size than the virtual object illustrated in FIG. 9b in the immersive view state as illustrated in FIG. 9c when the immersion score becomes 8 points.For example, a wearable device (200) according to one embodiment of the present disclosure may display a first application execution screen (530) (e.g., an active object) by enlarging the first application execution screen (530) by a specified ratio or more than the initially set size (e.g., the size of the screen displayed when the immersion level is 1), and by relatively darkening or blurring the area around the first application execution screen (530) (e.g., by applying a first visual effect (540) or by switching to a virtual reality environment).

[0133] FIG. 10 is a drawing for explaining a function or operation in which a display method of a virtual object is changed according to an increase or reset of an immersion score when there are multiple virtual objects according to one embodiment of the present disclosure.

[0134] FIGS. 11A to 11E are drawings for explaining the functions or operations described in FIG. 10 according to various embodiments of the present disclosure from a user interface perspective.

[0135] Referring to FIG. 10, a wearable device (200) according to an embodiment of the present disclosure may, in operation 1010, display execution screens of a plurality of applications as virtual objects based on a first immersion level. As illustrated in FIG. 11A, the wearable device (200) according to an embodiment of the present disclosure may obtain continuous user input for a first icon (510) (e.g., a gallery application) and a second icon (1110) (e.g., a calendar application). The user input according to an embodiment of the present disclosure may include a user's virtual touch on an icon and / or a user's gaze on an icon. According to one embodiment of the present disclosure, when the immersion level (e.g., the first immersion level as the initial level) is identified as, for example, level 3, the wearable device (200) may display execution screens of multiple applications (e.g., the first application execution screen (530) and the second application execution screen (1120)) and / or the tray (520) based on the immersion level 3, as illustrated in FIG. 11B. According to one embodiment of the present disclosure, when the immersion level is identified as 3, the wearable device (200) may display execution screens of multiple applications (e.g., the first application execution screen (530) and the second application execution screen (1120)) side by side, and may display the tray (520) in a blurred or darkened manner.

[0136] A wearable device (200) according to one embodiment of the present disclosure can, in operation 1020, identify the occurrence of a user interaction with a first execution screen among execution screens of a plurality of applications. For example, a wearable device (200) according to one embodiment of the present disclosure can identify that the user's gaze is continuously maintained on the first application execution screen (530) for a specified period of time or longer.

[0137] According to one embodiment of the present disclosure, the wearable device (200) may determine a second immersion level at operation 1030 based on the identification of the occurrence of the interaction at operation 1020. According to one embodiment of the present disclosure, the wearable device (200) may change the display properties of the first execution screen based on the second immersion level at operation 1040. According to one embodiment of the present disclosure, if the wearable device (200) identifies that the user's gaze is maintained on the first application execution screen (530) for a specified time or longer, the wearable device (200) may increase the immersion score by 1 point for each specified time from the immersion score corresponding to the first immersion level. Accordingly, the wearable device (200) according to one embodiment of the present disclosure can, for example, when the immersion score increases from 3 to 5, display the first application execution screen (530) by enlarging the size by a specified ratio from the initially set size, as illustrated in FIG. 11C, and can also move the display position so that the first application execution screen (530) is displayed at the substantial center of the field of view (FoV) of the wearable device (200). In this case, the display position of the second application execution screen (1120) according to one embodiment of the present disclosure can be changed, as illustrated in FIG. 11C, and its size can also be displayed by being reduced by a specified ratio. The wearable device (200) according to one embodiment of the present disclosure can display the first application execution screen (530) in an immersive view state, as illustrated in FIG. 11D, when the immersion score increases from 6 to 6, as the user's gaze is maintained on the first application execution screen (530).For example, a wearable device (200) according to one embodiment of the present disclosure may display a first application execution screen (530) (e.g., an active object) by enlarging the first application execution screen (530) by a specified ratio or more than the initially set size (e.g., the size of the screen displayed when the immersion level is 1) and relatively darkening or blurring the area around the first application execution screen (530) (e.g., by applying a first visual effect (540) or switching to a virtual reality environment).

[0138] According to an embodiment of the present disclosure, a wearable device (200) may determine, in operation 1050, whether a reset condition for a second immersion level has occurred. According to an embodiment of the present disclosure, for example, a wearable device (200) may identify that an external object (e.g., another user) is in proximity to a user. If the wearable device (200) according to an embodiment of the present disclosure identifies that an external object is in proximity to a user, the wearable device (200) may display application execution screens based on a first immersion level (e.g., level 3) or display application execution screens in a state of immersion level 1. FIG. 11E illustrates an example in which, by way of example, a wearable device (200) according to an embodiment of the present disclosure displays application execution screens based on the first immersion level if it identifies that an external object is in proximity to a user. However, according to an embodiment of the present disclosure, if it identifies that an external object is in proximity to a user, a virtual object may be displayed based on level 1.

[0139] According to one embodiment of the present disclosure, the wearable device (200) may display the first application execution screen (530) based on the display properties changed according to operation 1040, if the external object is not identified as being close to the user in operation 1060. For example, the wearable device (200) according to one embodiment of the present disclosure may display the first application execution screen (530) in an immersive view state.

[0140] FIG. 12 is a drawing for explaining a function or operation in which the display method of a virtual object is maintained or changed according to an increase in the immersion score when an application running through a wearable device (200) according to one embodiment of the present disclosure is a third type application (e.g., an application for initiation work).

[0141] FIGS. 13A to 13D are exemplary drawings for explaining the functions or operations described in FIG. 12 according to various embodiments of the present disclosure from a user interface perspective.

[0142] Referring to FIG. 12, a wearable device (200) according to an embodiment of the present disclosure may, in operation 1210, display a screen being displayed on an external electronic device as a virtual object based on a first immersion level. In order to determine the first immersion level, the wearable device (200) according to an embodiment of the present disclosure may identify real objects (e.g., a chair (507), a monitor (508), a desk lamp (509)) located around the user, as illustrated in FIG. 13A. The wearable device (200) according to an embodiment of the present disclosure may calculate an immersion score as 3 points when the surrounding environment is a work environment or the user's posture is in a sitting state. The wearable device (200) according to an embodiment of the present disclosure may determine an immersion level corresponding to an immersion score of 3 points as 3 levels. Accordingly, the wearable device (200) according to one embodiment of the present disclosure can display virtual objects according to three levels. As illustrated in FIG. 13B, the wearable device (200) according to one embodiment of the present disclosure can establish a communication session with the external electronic device when it is identified that the user is looking at the external electronic device while seated. To this end, the wearable device (200) according to one embodiment of the present disclosure and the external electronic device (e.g., a desktop computer including a monitor (508)) may be connected to the same server, or information that can be connected to operate with the external electronic device may be stored in the wearable device (200). The wearable device (200) according to one embodiment of the present disclosure and the external electronic device may be connected via a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association). A wearable device (200) according to one embodiment of the present disclosure can obtain information on an application currently running on an external electronic device from an external electronic device.According to one embodiment of the present disclosure, a wearable device (200) can display a screen being displayed on an external electronic device as a virtual object (e.g., a PC screen (1310)) using the acquired information, as illustrated in FIG. 13B . According to one embodiment of the present disclosure, a wearable device (200) can identify the type of application running on the external electronic device. For example, according to one embodiment of the present disclosure, a wearable device (200) can identify an application running on the external electronic device as an application for an initiation task.

[0143] According to an embodiment of the present disclosure, a wearable device (200) may determine a second immersion level based on identification of an input from an input device operable with an external electronic device at operation 1220. According to an embodiment of the present disclosure, a wearable device (200) may display a virtual object based on the second immersion level determined at operation 1220 at operation 1230. According to an embodiment of the present disclosure, a wearable device (200) may identify that a user input related to an application is received through an input device (e.g., a physical keyboard (508a) or a virtual keyboard). In this case, the wearable device (200) according to an embodiment of the present disclosure may determine such user input as frequent interaction with an immersive application and may not increase an immersion score. Accordingly, the wearable device (200) according to one embodiment of the present disclosure can maintain the current display state as shown in FIG. 13C. Alternatively, the wearable device (200) according to one embodiment of the present disclosure can determine such user input as frequent interaction with an experiential application and increase the immersion score. Accordingly, the wearable device (200) according to one embodiment of the present disclosure can display a virtual object while increasing the immersion level, as shown in FIG. 13D. However, in this case, the wearable device (200) according to one embodiment of the present disclosure can change the maximum immersion level from level 10 to level 7 when the application type is identified as the third type. Accordingly, the phenomenon of the virtual object growing in a way that does not match the user's intention can be prevented.

[0144] FIG. 14 is a drawing for explaining a function or operation of a wearable device (200) according to one embodiment of the present disclosure to change the display properties of at least one virtual object and interface as the immersion level increases.

[0145] FIGS. 15A and 15B are drawings for explaining the functions or operations described in FIG. 14 according to various embodiments of the present disclosure from a user interface perspective.

[0146] Referring to FIG. 14, a wearable device (200) according to an embodiment of the present disclosure may, in operation 1410, identify a change from a first immersion level to a second immersion level based on an interaction between a user and at least one virtual object. For example, the wearable device (200) according to an embodiment of the present disclosure may identify that the immersion level increases from 3 to 4 as the user's gaze is maintained on a designated application execution screen.

[0147] According to one embodiment of the present disclosure, the wearable device (200) may determine, in operation 1420, whether an interface (e.g., the tray (520) and / or the second application execution screen (1120)) is displayed among the virtual objects. If the wearable device (200) according to one embodiment of the present disclosure determines that the interface (e.g., the tray (520) and / or the second application execution screen (1120)) is displayed among the virtual objects, in operation 1430, the wearable device (200) may change the properties of the interface and the display properties of at least one virtual object based on a change to a second immersion level. For example, as illustrated in FIG. 15A, the wearable device (200) may darken or blur the tray (520) and darken or blur the second application execution screen (1120) as the immersion level increases. Alternatively, the wearable device (200) according to one embodiment of the present disclosure may not display the tray (520) or may display it darkly or blurrily while reducing the size of the second application execution screen (1120) as the immersion level increases, for example, as illustrated in FIG. 15b.

[0148] According to one embodiment of the present disclosure, if it is determined that an interface (e.g., a tray (520) and / or a second application execution screen (1120)) is not displayed among the virtual objects, the wearable device (200) may change the display properties of at least one virtual object based on the second immersion level in operation 1440. For example, according to one embodiment of the present disclosure, the wearable device (200) may display the first application execution screen (530) based on the second immersion level.

[0149] FIG. 16 is a drawing for explaining a function or operation of a wearable device (200) according to one embodiment of the present disclosure to change and display the display properties of grouped virtual objects based on an interaction with at least one virtual object among grouped virtual objects.

[0150] FIGS. 17A and 17B are drawings for explaining the functions or operations illustrated in FIG. 16 according to various embodiments of the present disclosure from a user interface perspective.

[0151] Referring to FIG. 16, a wearable device (200) according to an embodiment of the present disclosure may, in operation 1610, group execution screens of a plurality of applications (e.g., a third application execution screen (1710), a fourth application execution screen (1720), a fifth application execution screen (1730)) and display them as virtual objects based on a first immersion level. Displaying them in a grouped manner may mean, for example, that the execution screens of a plurality of applications (e.g., a third application execution screen (1710), a fourth application execution screen (1720), a fifth application execution screen (1730)) are controlled at once according to a user gesture on a control bar set to control the execution screens of a plurality of applications at once. A function or operation of identifying a user gesture according to an embodiment of the present disclosure may be performed by a gesture tracker (1936). Referring to FIG. 17a, an example is shown in which three application execution screens (e.g., a third application execution screen (1710), a fourth application execution screen (1720), and a fifth application execution screen (1730)) are grouped and displayed as virtual objects.

[0152] According to one embodiment of the present disclosure, a wearable device (200) can identify, in operation 1620, the occurrence of an interaction on at least one execution screen (e.g., the fourth application execution screen (1720)) among execution screens of a plurality of grouped applications (e.g., a third application execution screen (1710), a fourth application execution screen (1720), a fifth application execution screen (1730)). According to one embodiment of the present disclosure, a wearable device (200) can identify a user's gaze on the fourth application execution screen (1720).

[0153] According to an embodiment of the present disclosure, a wearable device (200) may, in operation 1630, group a plurality of application execution screens (e.g., a third application execution screen (1710), a fourth application execution screen (1720), and a fifth application execution screen (1730)) according to a second immersion level and display them as virtual objects based on the identification of the occurrence of an interaction. According to an embodiment of the present disclosure, when the application execution screens are grouped, if a change in the immersion level occurs for one of the application execution screens, the wearable device (200) may apply the same change in the immersion level to the other remaining execution screens. Referring to FIG. 17B, an embodiment is exemplarily illustrated in which the sizes of the third application execution screen (1710) and the fifth application execution screen (1730) are also enlarged and displayed according to the occurrence of an interaction for the fourth application execution screen (1720).

[0154] FIG. 17C is a diagram illustrating a function or operation of a wearable device (200) according to an embodiment of the present disclosure to display a virtual object on a full virtual reality based on a first immersion level exceeding a threshold level. The wearable device (200) according to an embodiment of the present disclosure may display a virtual object on a full virtual reality (540) without displaying the virtual object in a pass-through manner when the identified first immersion level exceeds a threshold level (e.g., level 5).

[0155] FIG. 18A and FIG. 18B are drawings illustrating another type of wearable device according to various embodiments of the present disclosure.

[0156] Referring to FIGS. 18A and 18B , in one embodiment, camera modules (1811, 1812, 1813, 1814, 1815, 1816) and / or a depth sensor (1817) may be disposed on a first surface (1810) of the housing to obtain information related to the surrounding environment of the wearable device (200). In one embodiment, the camera modules (1811, 1812) may obtain images related to the surrounding environment of the wearable device. In one embodiment, the camera modules (1813, 1814, 1815, 1816) may obtain images while the wearable device (200) is worn by a user. The images acquired through the camera modules (1813, 1814, 1815, 1816) can be used for simultaneous localization and mapping (SLAM), 6 degrees of freedom (6DoF), 3 degrees of freedom (3DoF), object recognition, and / or tracking, and can be used as input for a wearable electronic device by recognizing and / or tracking a user's hand. In one embodiment, a depth sensor (1817) can be configured to transmit a signal and receive a signal reflected from a subject, and can be used for the purpose of confirming the distance to an object, such as time of flight (TOF). According to one embodiment, a camera module (1825, 1826) for face recognition and / or a display (1821) (and / or a lens) can be arranged on the second side (1820) of the housing. In one embodiment, a camera module (1825, 1826) for facial recognition adjacent to the display may be used to recognize a user's face, or may recognize and / or track both eyes of the user. In one embodiment, the display (1821) (and / or lens) may be disposed on a second side (1820) of the wearable device (200).In one embodiment, the wearable device (200) may not include camera modules (1815, 1816) among the plurality of camera modules (1813, 1814, 1815, 1816). Although not illustrated in FIGS. 18A and 18B , the wearable device (200) may further include at least one of the configurations illustrated in FIGS. 2A to 2C . As described above, the wearable device (200) according to one embodiment may have a form factor for being worn on a user's head. The wearable device (200) may further include a strap for being fixed on a body part of the user, and / or a wearing member (e.g., a wearing member (203)). The wearable device (200) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0157] FIG. 19 is a block diagram illustrating a wearable device (200) according to one embodiment of the present disclosure.

[0158] Referring to FIG. 19, a wearable device (200) according to an embodiment of the present disclosure may include a processor (120). A wearable device (200) according to an embodiment of the present disclosure may include a display (1910) (e.g., a display module (160)). A wearable device (200) according to an embodiment of the present disclosure may include a sensor (1920) (e.g., a sensor module (176)). A sensor (1920) according to an embodiment of the present disclosure may include at least one of an image sensor (1922) for photographing a user's face or the real world, an eye sensor (1924) for tracking a user's gaze, and an audio sensor (1926) for outputting auditory information. A wearable device (200) according to an embodiment of the present disclosure may include a communication module (190).

[0159] A wearable device (200) according to one embodiment of the present disclosure may include a memory (130). At least one of user information recognition logic (1930), external environment recognition logic (1940), and user task recognition logic (1950) may be stored in the memory (130) according to one embodiment of the present disclosure. The user information recognition logic (1930) according to one embodiment of the present disclosure may include at least one of a posture recognizer (1932) for recognizing a user's posture, a gaze tracker (1934) for tracking a user's gaze, and a gesture tracker (1936) for tracking a user's gesture. The external environment recognition logic (1940) according to one embodiment of the present disclosure may include at least one of object recognition logic (1942) for recognizing an object in the real world, location information recognition logic (1944) for recognizing the location of the wearable device (200), and external device recognition logic (1946) for recognizing an external device located around the wearable device (200). The user task recognition logic (1950) according to one embodiment of the present disclosure may include at least one of application characteristic determination logic (1952) for determining the type of application, user interaction recognition logic (1954) for determining the interaction between the user and the application execution screen, and user interrupt recognition logic (1956) for identifying the proximity of an external object. The memory (130) according to one embodiment of the present disclosure may store immersion level determination logic (1960) for determining an immersion level based on an immersion score. In one embodiment of the present disclosure, a memory (130) may store immersion level reflection logic (1970) for executing an application in which the immersion level is reflected.

[0160] A wearable device (200) according to one embodiment of the present disclosure may include a sensor (1920), which may include an image sensor (1922) for acquiring at least one image, an eye sensor (1924) for tracking a user's eyes, and an audio sensor (1926) for outputting auditory information. A wearable device (200) according to one embodiment of the present disclosure may include a display (1910), a processor (120), and / or a communication module (190). A wearable device (200) according to one embodiment of the present disclosure includes at least one sensor (e.g., a sensor module (176) of FIG. 1), at least one camera (e.g., a second camera module (253)), at least one processor (e.g., a processor (120) of FIG. 1), and a memory, wherein the memory includes instructions that, when executed, cause the at least one processor to identify a real object located around a user based on an image of the real world acquired through the at least one camera, identify a posture of the user wearing the wearable device based on sensed data acquired by the at least one sensor, determine a first immersion level based on the identified real object and the posture of the user, display an execution screen of at least one application as a virtual object based on the determined first immersion level, determine a second immersion level based on an interaction with the virtual object and the user while the virtual object is displayed, and change a display method of the virtual object based on the determined second immersion level. It can be set to save.

[0161] According to one embodiment of the present disclosure, the execution screen displayed according to the first immersion level may be displayed in virtual reality.

[0162] According to one embodiment of the present disclosure, the interaction may include a state in which the user's gaze is maintained on the execution screen of the at least one application for a specified period of time or longer.

[0163] According to one embodiment of the present disclosure, the interaction may include a state in which a user input for performing a task related to the at least one application is maintained for a specified period of time or longer.

[0164] According to one embodiment of the present disclosure, the execution screen displayed according to the second immersion level may include a screen having a size substantially larger than that of the execution screen displayed according to the first immersion level.

[0165] According to one embodiment of the present disclosure, the execution screen displayed according to the second immersion level may include a screen in which the brightness of the periphery of the area where the execution screen is displayed is displayed brighter than the execution screen.

[0166] According to one embodiment of the present disclosure, the execution screen displayed according to the second immersion level may include a screen displayed at a position closer to the center area of ​​the field of view of the wearable device than the position of the execution screen displayed according to the first immersion level.

[0167] Electronic devices according to various embodiments disclosed in the present disclosure may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of the present disclosure are not limited to the aforementioned devices.

[0168] The various embodiments of the present disclosure and the terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. In the present disclosure, each of the phrases "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among the phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0169] The term "module" used in various embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0170] Various embodiments of the present disclosure may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0171] According to one embodiment, the method according to various embodiments disclosed in the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0172] According to one embodiment of the present disclosure, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separately arranged in other components. According to one embodiment of the present disclosure, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment of the present disclosure, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0173] While the present disclosure has been shown and described with reference to various embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. In wearable devices, At least one sensor, At least one camera, memory for storing one or more programs, and comprising at least one sensor, at least one camera, and a processor communicatively coupled to the memory; The one or more programs comprise computer-executable instructions which, when individually or collectively executed by the processor, cause the wearable device to: Identifying real objects located around the user based on images of the real world acquired through at least one camera; Based on the sensing data acquired by the at least one sensor, the posture of the user wearing the wearable device is identified, Based on the identified real object and the user's posture, a first immersion level is determined, Based on the first immersion level determined above, display the execution screen of at least one application as a virtual object, While the above virtual object is displayed, a second immersion level is determined based on the interaction with the virtual object and the user, A wearable device characterized in that it is set to store instructions for changing the display method of the virtual object based on the determined second immersion level.

2. In paragraph 1, A wearable device, characterized in that the execution screen displayed according to the first immersion level is displayed in virtual reality.

3. In paragraph 1 or 2, A wearable device, characterized in that the above interaction includes a state in which the user's gaze is maintained on the execution screen of at least one application for a specified period of time or longer.

4. In any one of paragraphs 1 to 3, A wearable device, characterized in that the above interaction includes a state in which a user input for performing a task related to at least one application is maintained for a specified period of time or longer.

5. In any one of paragraphs 1 to 4, A wearable device, characterized in that the execution screen displayed according to the second immersion level includes a screen having a size substantially larger than that of the execution screen displayed according to the first immersion level.

6. In any one of paragraphs 1 to 5, A wearable device, characterized in that the execution screen displayed according to the second immersion level includes a screen in which the brightness of the periphery of the area where the execution screen is displayed is displayed darker than the execution screen.

7. In any one of paragraphs 1 to 6, A wearable device, characterized in that the execution screen displayed according to the second immersion level includes a screen displayed at a position closer to the center area of ​​the field of view of the wearable device than the position of the execution screen displayed according to the first immersion level.

8. In the method, An operation of identifying a real object located around the user based on an image of the real world acquired through at least one camera of the wearable device, An operation of identifying a posture of a user wearing the wearable device based on sensing data acquired by at least one sensor of the wearable device, An action for determining a first immersion level based on the identified real object and the user's posture, An operation of displaying the execution screen of at least one application as a virtual object based on the first immersion level determined above, An operation for determining a second immersion level based on the interaction between the virtual object and the user while the virtual object is displayed, and A method characterized by including an action of changing a display method of the virtual object based on the determined second immersion level.

9. In a computer-readable non-transitory recording medium, The above computer-readable non-transitory recording medium is configured to store a plurality of instructions, which, when executed, cause a processor of the electronic device to: Identifying real objects located around the user based on images of the real world acquired through at least one camera of the wearable device, Identifying the posture of the user wearing the wearable device based on sensing data acquired by at least one sensor of the wearable device, Based on the identified real object and the user's posture, a first immersion level is determined, Based on the first immersion level determined above, display the execution screen of at least one application as a virtual object, While the above virtual object is displayed, a second immersion level is determined based on the interaction with the virtual object and the user, A non-transitory recording medium characterized by being set to store instructions for changing the display method of the virtual object based on the determined second immersion level.

10. In paragraph 9, A non-transitory recording medium, characterized in that the execution screen displayed according to the first immersion level is displayed in virtual reality.

11. In clause 9 or 10, A non-transitory recording medium, characterized in that the above interaction includes a state in which the user's gaze is maintained on the execution screen of at least one application for a specified period of time or longer.

12. In any one of paragraphs 9 to 11, A non-transitory recording medium, characterized in that the above interaction includes a state in which a user input for performing a task related to at least one application is maintained for a specified period of time or longer.

13. In any one of paragraphs 9 to 12, A non-transitory recording medium, characterized in that the execution screen displayed according to the second immersion level includes a screen of a size substantially larger than the size of the execution screen displayed according to the first immersion level.

14. In any one of paragraphs 9 to 13, A non-transitory recording medium, characterized in that the execution screen displayed according to the second immersion level includes a screen in which the brightness of the periphery of the area where the execution screen is displayed is displayed brighter than the execution screen.

15. In any one of paragraphs 9 to 14, A non-transitory recording medium, characterized in that the action of changing the display method of the virtual object includes changing the action of displaying the virtual object in a virtual reality environment that does not include the real world from the action of simultaneously displaying the virtual object and the real world.

Images