Wearable electronic device and operation method thereof

Abstract

A wearable electronic device, according to an embodiment of the present disclosure, may comprise: a display that displays an external real object and a virtual object on a single screen; a display driving unit that controls driving of the display; an audio module that converts electrical signals into sound and outputs the sound; at least one processor that controls the operations of the display driving unit and the audio module; and a memory that stores instructions. When the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may display the virtual object at a first position spaced a first distance apart from a reference point in a first direction. When the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may set a current location of the wearable electronic device as the reference point. When the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may provide information of the virtual object to a user via voice when the user's line of sight at the current location is in a direction in which the virtual object cannot be viewed. Various other embodiments are also possible.

Description

Wearable electronic device and method of operation thereof

[0001] The present disclosure relates to a wearable electronic device capable of providing interaction with a virtual object according to the position and gaze of a user wearing the wearable electronic device (e.g., AR electronic device, VR electronic device) and a method of operating the same.

[0002] Portable electronic devices (e.g., wearable electronic devices) that provide various functions are being developed to satisfy consumers' purchasing desires. With the advancement of display technology, wearable electronic devices including displays capable of implementing augmented reality (AR) (e.g., AR glasses, HMDs) have been developed. Wearable electronic devices including displays capable of implementing augmented reality can combine AR images displayed on the screen with real scenes (e.g., external images, external real photos) to be perceived by the user's eyes. Users can simultaneously view AR images displayed on the screen and real scenes (e.g., external images, external real photos), and can experience augmented reality by combining AR images with real scenes (e.g., external images, external real photos).

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] A wearable electronic device operates by being worn on a user's face (or head), and can capture real objects through a camera and display virtual objects along with the real objects on a display. When a virtual object in a world-locked state is displayed while a user is wearing a wearable electronic device (e.g., HMD), the user may not accurately perceive the virtual object if the distance between the user and the virtual object is too close or too far. Additionally, the user may not accurately perceive the virtual object if the user and the virtual object overlap. Furthermore, the user may not accurately perceive the virtual object if the user's gaze direction differs from the location where the virtual object is displayed. Additionally, the user may not accurately perceive the virtual object if the real object and the virtual object overlap.

[0005] Embodiments of the present disclosure may provide a wearable electronic device capable of outputting information of a virtual object as voice (e.g., sound) when the distance between a user and a virtual object is too close or too far apart, and a method of operating the same.

[0006] Embodiments of the present disclosure may provide a wearable electronic device capable of changing and displaying the position of a virtual object when a user and a virtual object overlap, and a method of operating the same.

[0007] Embodiments of the present disclosure may provide a wearable electronic device capable of outputting information of a virtual object as voice (e.g., sound) when a user and a virtual object overlap, and a method of operating the same.

[0008] Embodiments of the present disclosure may provide a wearable electronic device capable of outputting information of an object as voice (e.g., sound) when the user's gaze direction differs from the location where the virtual object is displayed, and a method of operating the same.

[0009] Embodiments of the present disclosure may provide a wearable electronic device capable of changing and displaying the position of a virtual object when a real object and a virtual object overlap, and a method of operating the same.

[0010] Embodiments of the present disclosure may provide a wearable electronic device capable of outputting information of a virtual object as voice (e.g., sound) when a real object and a virtual object overlap, and a method of operating the same.

[0011] Embodiments of the present disclosure may provide a wearable electronic device capable of dividing a virtual object into a plurality of parts and displaying it so that the user and the virtual object do not overlap when a real object and a virtual object overlap, and a method of operating the same.

[0012] The technical tasks intended to be accomplished in this document are not limited to those mentioned above, and other technical tasks not mentioned can be clearly understood by a person skilled in the art to which this document belongs from the description below.

[0013] A wearable electronic device according to an embodiment of the present disclosure may include a display that displays an external real object and a virtual object on a single screen, a display driver that controls the driving of the display, an audio module that converts an electrical signal into sound and outputs it, at least one processor that controls the operation of the display driver and the audio module, and a memory that stores instructions. When the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may display a virtual object at a first location spaced a first distance in a first direction from a reference point. When the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may set the local location of the wearable electronic device as a reference point. When the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may provide information about the virtual object to the user in voice if the user's gaze from the current location is in a direction where the virtual object cannot be seen.

[0014] A method of operating a wearable electronic device according to one embodiment of the present disclosure displays an external actual object and a virtual object on a single screen on a display, displays the virtual object at a first position spaced a first distance in a first direction from a reference point, sets the local position of the wearable electronic device as a reference point, and when the user's gaze from the current position is in a direction where the virtual object cannot be seen, provides information about the virtual object to the user in voice.

[0015] The wearable electronic device and the method of operation thereof according to the embodiment of the present disclosure can improve the safety of use and user experience of the wearable electronic device by changing and displaying the size, position, and shape of a virtual object.

[0016] The wearable electronic device and the method of operation thereof according to the embodiment of the present disclosure can enable a user to perceive information about a virtual object without missing it.

[0017] The wearable electronic device and the method of operation according to the embodiment of the present disclosure can reduce visual clutter in a virtual space and increase space utilization by moving the position of a virtual object according to the distance between the user's gaze direction and the virtual object.

[0018] A wearable electronic device and a method of operating the same according to an embodiment of the present disclosure can reduce resources required for information transmission by summarizing meaningful information among information of a plurality of virtual objects and providing summary information of virtual objects to a user.

[0019] A wearable electronic device and a method of operation thereof according to an embodiment of the present disclosure can output information of a virtual object as voice (e.g., sound) when the distance between the user and the virtual object is too close or too far apart.

[0020] The wearable electronic device and the method of operation thereof according to the embodiment of the present disclosure can change and display the position of a virtual object when a user and a virtual object overlap.

[0021] A wearable electronic device and a method of operation thereof according to an embodiment of the present disclosure can output information of a virtual object as voice (e.g., sound) when a user and a virtual object overlap.

[0022] A wearable electronic device and a method of operation thereof according to an embodiment of the present disclosure can output information of an object as voice (e.g., sound) when the user's gaze direction is different from the location where a virtual object is displayed.

[0023] The wearable electronic device and the method of operation thereof according to the embodiment of the present disclosure can change and display the position of a virtual object when a real object and a virtual object overlap.

[0024] A wearable electronic device and a method of operation thereof according to an embodiment of the present disclosure can output information of a virtual object as voice (e.g., sound) when a real object and a virtual object overlap.

[0025] The wearable electronic device and the method of operation thereof according to an embodiment of the present disclosure can display a virtual object by dividing it into a plurality of parts so that the user and the virtual object do not overlap when a real object and a virtual object overlap.

[0026] In addition, various effects identified directly or indirectly through this document may be provided.

[0027] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0028] In relation to the description of the drawings, the same (or similar) reference numerals may be used to describe identical (or similar) components, features, and structures.

[0029] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment of the present disclosure.

[0030] FIG. 2a shows an example of a perspective view of a wearable electronic device.

[0031] FIG. 2b shows an example of one or more hardware components placed within a wearable electronic device.

[0032] FIGS. 3a and 3b show examples of the appearance of a wearable electronic device.

[0033] Figure 4 shows an example of a block diagram of a wearable electronic device.

[0034] Figure 5 shows an example of a block diagram of an electronic device for displaying an image in virtual space.

[0035] Figure 6 illustrates an example of a structure of multiple layers.

[0036] FIG. 7a is a block diagram of a wearable electronic device according to an embodiment of the present disclosure.

[0037] FIG. 7b is a drawing illustrating a method of operation of a wearable electronic device according to an embodiment of the present disclosure.

[0038] FIG. 8 relates to a method of operation of a wearable electronic device according to an embodiment of the present disclosure, and is a drawing showing the recognition of the distance and direction between a user and a virtual object and the display of a virtual object by changing the virtual object.

[0039] FIG. 9 relates to a method of operation of a wearable electronic device according to an embodiment of the present disclosure, and is a drawing showing the recognition of the distance and direction between a user and a virtual object and the display of a virtual object by changing the virtual object.

[0040] FIG. 10 illustrates the operation of a wearable electronic device according to an embodiment of the present disclosure, and is a drawing showing how to change and display a virtual object when a virtual object and a real object overlap.

[0041] FIG. 11 illustrates the operation of a wearable electronic device according to an embodiment of the present disclosure, and is a drawing showing the recognition of the distance and direction between a user and a virtual object and the display of a changed virtual object.

[0042] FIG. 12 is a drawing showing the operation of a wearable electronic device according to an embodiment of the present disclosure.

[0043] FIG. 13 illustrates the operation of a wearable electronic device according to an embodiment of the present disclosure, showing that information of a virtual object is output as voice when the direction of the user's gaze is at an angle where the virtual object cannot be seen.

[0044] FIG. 14 illustrates the operation of a wearable electronic device according to an embodiment of the present disclosure, and is a diagram showing that information of a virtual object is output as voice when a user and a virtual object overlap.

[0045] FIG. 15 illustrates the operation of a wearable electronic device according to an embodiment of the present disclosure, and is a diagram showing that information of a virtual object is output as voice when the user cannot look at the virtual object.

[0046] FIG. 16 is a drawing showing the operation of a wearable electronic device according to an embodiment of the present disclosure.

[0047] FIG. 17 is a drawing showing the operation of a wearable electronic device according to an embodiment of the present disclosure.

[0048] FIG. 18 is a drawing showing the operation of a wearable electronic device according to an embodiment of the present disclosure.

[0049] It should be noted that throughout the drawings, the same reference number is used to describe the same or similar elements, features, and structures.

[0050] The following description, with reference to the attached drawings, is provided to facilitate a comprehensive understanding of various embodiments of the disclosure as defined by the claims and their equivalents. While various specific details are included to aid understanding, they should be considered merely illustrative. Accordingly, those skilled in the art will recognize that various changes and modifications to the various embodiments described herein may be made without departing from the scope and spirit of the disclosure. Additionally, for clarity and brevity, descriptions of well-known functions and configurations may be omitted.

[0051] The terms and words used in the following description and claims are not limited to their literary meanings and are merely used by the applicant to enable a clear and consistent understanding of this document. Accordingly, it should be apparent to those skilled in the art that the following description of various embodiments of this document is provided for illustrative purposes only and is not intended to limit this document as defined by the appended claims and their equivalents.

[0052] The singular form should be understood to include plural referents unless the context clearly indicates otherwise. Thus, for example, a reference to "component surfaces" may include a reference to one or more of such surfaces.

[0053] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment of the present disclosure.

[0054] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0055] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0056] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0057] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0058] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0059] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0060] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0061] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0062] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0063] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0064] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0065] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0066] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0067] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0068] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0069] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0070] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0071] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0072] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0073] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0074] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0075] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0076] An electronic device according to one embodiment disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable electronic device, or a consumer electronics device. The electronic device according to the embodiment of this document is not limited to the aforementioned devices.

[0077] The embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any component (e.g., the first) is referred to as "coupled" or "connected" to another component (e.g., the second), with or without the terms "functionally" or "communicationally," it means that said component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0078] As used in one embodiment of the present disclosure, the term “module” may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0079] One embodiment of the present disclosure may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0080] According to one embodiment, the method according to one embodiment disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0081] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0082] In the various embodiments of the present disclosure described below, a hardware-based approach is described as an example. However, since the various embodiments of the present disclosure include techniques using both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0083] Terms used in the following description to refer to images (e.g., image, frame, camera frame, captured image, camera image), terms referring to a user's hand (e.g., hand object, candidate object, hand candidate object, bounding box, candidate hand object), terms referring to signals (e.g., signaling, control signal, data, control data, request signal, information), terms referring to locations (e.g., location information, area information, object information, object location, object coordinates, reference object, coordinate information, location, coordinate, relative coordinate, absolute coordinate, coordinate system), terms referring to values ​​(e.g., threshold, reference value, reference area, reference range, level, threshold level, threshold, range, value, area), terms for operation states (e.g., step, operation, procedure), terms referring to network entities, terms referring to device components, etc., are provided as examples for the convenience of explanation. Accordingly, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings may be used. Additionally, terms such as '...part', '...device', '...object', '...body' used below may refer to at least one shape structure or a unit that processes a function.

[0084] Additionally, in this disclosure, expressions of "greater than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled; however, this is merely for the purpose of expressing an example and does not exclude descriptions of "greater than" or "less than." Conditions described as "greater than" may be replaced with "greater than," conditions described as "less than" may be replaced with "less than," and conditions described as "greater than and less than" may be replaced with "greater than and less than." Furthermore, "A" to "B" below refer to at least one of elements from A (including A) to B (including B). Below, "C" and / or "D" refers to including at least one of "C" or "D," i.e., {"C", "D", "C" and "D"}.

[0085] In embodiments of the present disclosure, an electronic device for displaying an image in a virtual space may be a wearable electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (101, 400) of FIG. 4, and the wearable electronic device (101, 400) of FIG. 7a). The wearable electronic device (101, 400) may include a head-mounted display (HMD) that is wearable on a user's head. The wearable electronic device (101, 400) may be referred to as a head-mount device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of a wearable electronic device (101, 400) having the form of glasses is shown, the embodiment is not limited thereto.

[0086] An example of a hardware configuration included in a wearable electronic device (101, 400) is described exemplarily with reference to FIG. 4 and FIG. 7a.

[0087] An example of the structure of a wearable electronic device (101, 400) that can be worn on a user's head is described with reference to FIG. 2a, FIG. 2b, FIG. 3a and / or FIG. 3b.

[0088] The wearable electronic device (101, 400) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) to be attached to a user's head to form an HMD.

[0089] According to one embodiment, a wearable electronic device (101, 400) can perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, while a user is wearing the wearable electronic device (101, 400), the wearable electronic device (101, 400) may include at least one lens positioned adjacent to the user's eye. The wearable electronic device (101, 400) may combine light emitted from a display of the wearable electronic device (101, 400) with ambient light passing through the lens. The display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable electronic device (101, 400) combines the ambient light and the light emitted from the display, the user can see an image in which a real object perceived by the ambient light and a virtual object formed by the light emitted from the display are mixed. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0090] According to one embodiment, a wearable electronic device (101, 400) can perform functions related to VST (video see-through or visible see-through) and / or virtual reality (VR). For example, while a user is wearing the wearable electronic device (101, 400), the wearable electronic device (101, 400) may include a housing that covers the user's eyes. The wearable electronic device (101, 400) may include a display disposed on a first surface of the housing facing the eyes while in the state. The wearable electronic device (101, 400) may include a camera disposed on a second surface opposite to the first surface. Using the camera, the wearable electronic device (101, 400) can acquire an image and / or video representing ambient light. A wearable electronic device (101, 400) may output the image and / or video within a display placed on the first surface, thereby allowing a user to perceive the ambient light through the display. A displaying area (or displaying region) (or active area or active region) of the display placed on the first surface may be formed by one or more pixels included in the display. The wearable electronic device (101, 400) may composite a virtual object with the image and / or video output through the display, thereby allowing a user to perceive the virtual object along with a real object perceived by the ambient light.

[0091] According to one embodiment, a wearable electronic device (101, 400) can identify or recognize the position or location and / or direction or orientation of the wearable electronic device (101, 400) based on an image (and / or video) obtained or acquired using a camera. The wearable electronic device (101, 400) can obtain information about the external space using one or more cameras and / or one or more sensors. The information may include a geographic location of the external space (e.g., GPS (global positioning system) coordinates) identified by one or more sensors. The information may include an image and / or video of the external space identified by one or more cameras. The wearable electronic device (101, 400) can perform object recognition on the image and / or video to identify external objects contained in the external space from the image and / or video.

[0092] Hereinafter, with reference to FIGS. 2a, FIGS. 2b, FIGS. 3a, FIGS. 3b, and FIGS. 4, an example of a hardware configuration of a wearable electronic device (101, 400) is described.

[0093] FIG. 2a illustrates an example of a perspective view of a wearable electronic device. FIG. 2b illustrates an example of one or more hardware components disposed within the wearable electronic device.

[0094] According to one embodiment, the wearable electronic device (101, 400) may have the form of glasses that are wearable on a part of a user's body (e.g., head). The wearable electronic device (101, 400) of FIGS. 2a and 2b may be an example of the wearable electronic device (101, 400) of FIG. 1. The wearable electronic device (101, 400) may include a head-mounted display (HMD). For example, the housing of the wearable electronic device (101, 400) may include a flexible material such as rubber and / or silicone that has a shape that adheres to a part of the user's head (e.g., a part of the face covering both eyes). For example, the housing of a wearable electronic device (101, 400) may include one or more straps that can be twined around a user's head, and / or one or more temples that can be attached to the ears of the head.

[0095] Referring to FIG. 2a, a wearable electronic device (101) according to one embodiment may include at least one display (250) and a frame (200) supporting at least one display (250).

[0096] According to one embodiment, a wearable electronic device (101) may be worn on a part of a user's body. The wearable electronic device (101) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable electronic device (101). For example, the wearable electronic device (101) may display a virtual reality image provided by at least one optical device (282, 284) of FIG. 2b on at least one display (250) in response to a specified gesture of the user obtained through the motion recognition camera (260-2, 260-3) of FIG. 2b.

[0097] According to one embodiment, at least one display (250) can provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at locations corresponding to the user's left eye and right eye, respectively.

[0098] Referring to FIG. 2b, at least one display (250) may provide visual information transmitted from external light to a user through a lens included in at least one display (250) and other visual information distinct from said visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) may include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area may be formed on the second surface (232) of at least one display (250). When a user wears the wearable electronic device (101), external light may be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided by at least one optical device (282, 284) on a real image transmitted through external light in a display area formed on the second surface (232).

[0099] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits it to a user. At least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed on the exterior or at least a portion of the interior of at least one waveguide (233, 234). The nano pattern may be formed based on a polygonal and / or curved grating structure. Light incident on one end of at least one waveguide (233, 234) may be propagated to the other end of at least one waveguide (233, 234) by the nano pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be placed within a wearable electronic device (101) to guide a screen displayed by at least one display (250) to the user's eye. For example, the screen may be transmitted to the user's eye based on total internal reflection (TIR) ​​occurring within at least one waveguide (233, 234).

[0100] According to one embodiment, a wearable electronic device (101) can analyze an object included in a real-world image collected through a shooting camera (260-4), combine a virtual object corresponding to an object among the analyzed objects that is the target of augmented reality provision, and display it on at least one display (250). The virtual object may include at least one of text and an image regarding various information related to the object included in the real-world image. The wearable electronic device (101) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable electronic device (101, 400) can perform spatial recognition (e.g., SLAM (simultaneous localization and mapping)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable electronic device (101) can view the image displayed on at least one display (250).

[0101] According to one embodiment, the frame (200) may be formed as a physical structure that allows the wearable electronic device (101) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that when the user wears the wearable electronic device (101), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) so that they are positioned corresponding to the user's left and right eyes.

[0102] Referring to FIG. 2a, the frame (200) may include an area (220) in which at least a portion comes into contact with a part of the user's body when the user wears the wearable electronic device (101). For example, the area (220) of the frame (200) in contact with a part of the user's body may include an area in which the wearable electronic device (101) comes into contact with a part of the user's nose, a part of the user's ear, and a part of the side of the user's face.

[0103] According to one embodiment, the frame (200) may include a nose pad (210) that contacts a part of the user's body. When the wearable electronic device (101) is worn by the user, the nose pad (210) may contact a part of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that contact another part of the user's body distinct from the part of the user's body.

[0104] For example, the frame (200) may include a first rim (201) covering at least a portion of a first display (250-1), a second rim (202) covering at least a portion of a second display (250-2), a bridge (203) positioned between the first rim (201) and the second rim (202), a first pad (211) positioned along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) positioned along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) extending from the first rim (201) and fixed to a portion of the wearer's ear, and a second temple (205) extending from the second rim (202) and fixed to a portion of the ear opposite to the first. The first pad (211) and the second pad (212) may come into contact with a part of the user's nose, and the first temple (204) and the second temple (205) may come into contact with a part of the user's face and a part of the ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through a first hinge unit (206) positioned between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through a second hinge unit (207) positioned between the second rim (202) and the second temple (205). According to one embodiment, a wearable electronic device (101, 400) can identify an external object touching the frame (200) (e.g., a user's fingertip) and / or a gesture performed by said external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame (200).

[0105] According to one embodiment, the wearable electronic device (101) may include hardware that performs various functions (e.g., hardware to be described later based on the block diagram of FIG. 4). For example, the hardware may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers (255-1, 255-2)), a microphone (e.g., microphones (265-1, 265-2, 265-3)), a light-emitting module (not shown), and / or a PCB (printed circuit board) (290) (e.g., a printed circuit board). The various hardware may be placed within a frame (200).

[0106] According to one embodiment, a microphone (e.g., microphones (265-1, 265-2, 265-3)) of a wearable electronic device (101) is positioned on at least a portion of a frame (200) to acquire a sound signal. A first microphone (265-1) positioned on a bridge (203), a second microphone (265-2) positioned on a second rim (202), and a third microphone (265-3) positioned on a first rim (201) are shown in FIG. 2b, but the number and position of the microphones (265) are not limited to the embodiment of FIG. 2b. When the number of microphones (265) included in the wearable electronic device (101) is two or more, the wearable electronic device (101, 400) can identify the direction of the sound signal by using a plurality of microphones positioned on different portions of the frame (200).

[0107] According to one embodiment, at least one optical device (282, 284) can project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be positioned adjacent to at least one display (250). At least one optical device (282, 284) may be included within at least one display (250) as part of at least one display (250).

[0108] According to one embodiment, the wearable electronic device (101) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) positioned at the edge of the first display (250-1) and a second optical device (284) positioned at the edge of the second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) positioned on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) positioned on the second display (250-2).

[0109] In one embodiment, the camera (260) may include a shooting camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 206-3). The shooting camera (260-4), the eye tracking camera (260-1), and the motion recognition camera (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position of the eyes or the gaze of a user wearing the wearable electronic device (101). For example, the wearable electronic device (101, 400) may detect the gaze from an image containing the user's pupils obtained through the eye tracking camera (260-1). A wearable electronic device (101) can identify an object focused by a user (e.g., a real object, and / or a virtual object) by using the user's gaze acquired through an eye-tracking camera (260-1). The wearable electronic device (101), having identified the focused object, can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable electronic device (101) can represent a portion corresponding to the eyes of an avatar representing the user in a virtual space by using the user's gaze acquired through the eye-tracking camera (260-1). The wearable electronic device (101) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first area related to the gaze within the image and the visual quality of a second area distinguished from the first area (e.g., resolution, brightness, saturation, grayscale, PPI) may differ from each other.The wearable electronic device (101) can acquire an image having a visual quality of a first region and a visual quality of a second region that matches the user's gaze by using foveated rendering. For example, if the wearable electronic device (101) supports an iris recognition function, user authentication can be performed based on iris information acquired using an eye-tracking camera (260-1). An example in which the eye-tracking camera (260-1) is positioned toward the user's right eye is shown in FIG. 2b, but the embodiment is not limited thereto, and the eye-tracking camera (260-1) may be positioned alone toward the user's left eye or toward both eyes.

[0110] In one embodiment, the camera (260-4) can capture a real image or background to be matched with a virtual image in order to implement augmented reality or mixed reality content. The camera (260-4) can be used to acquire high-resolution images based on HR (high resolution) or PV (photo video). The camera (260-4) can capture an image of a specific object located at the position viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information regarding a real image or background including the image of the specific object acquired using the camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable electronic device (101) can compensate for depth information (e.g., the distance between the wearable electronic device (101) and an external object acquired through a depth sensor) using the image acquired through the camera (260-4). The wearable electronic device (101) can perform object recognition through an image acquired using a shooting camera (260-4). The wearable electronic device (101) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the shooting camera (260-4). The wearable electronic device (101) can perform a pass-through function to superimpose an image acquired through the shooting camera (260-4) onto at least a portion of a screen representing a virtual space while displaying the screen representing a virtual space on at least one display (250). In one embodiment, the shooting camera (260-4) may be placed on a bridge (203) positioned between a first rim (201) and a second rim (202).

[0111] According to one embodiment, the eye tracking camera (260-1) can achieve more realistic augmented reality by tracking the gaze of a user wearing a wearable electronic device (101), thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the user looks straight ahead, the wearable electronic device (101) can naturally display environmental information related to the user's front at the location where the user is situated on at least one display (250). The eye tracking camera (260-1) may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the eye tracking camera (260-1) may receive a gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light.

[0112] In one embodiment, the eye-tracking camera (260-1) may be positioned at locations corresponding to the user's left and right eyes. For example, the eye-tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable electronic device (101) is located.

[0113] According to one embodiment, a motion recognition camera (260-2, 260-3) can provide a specific event to a screen provided on at least one display (250) by recognizing the movement of the user's entire body or part thereof, such as the user's torso, hands, or face. The motion recognition camera (260-2, 260-3) can recognize the user's gesture, acquire a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display (250). A processor can identify the signal corresponding to the gesture and, based on the identification, perform a designated function. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for a 6-degrees-of-freedom pose (6 dof pose). A processor can use the motion recognition camera (260-2, 260-3) to perform a gesture recognition function and / or an object tracking function. In one embodiment, a motion recognition camera (260-2, 260-3) may be placed on the first rim (201) and / or the second rim (202).

[0114] According to one embodiment, the camera (260) included in the wearable electronic device (101) is not limited to the eye-tracking camera (260-1) and motion recognition camera (260-2, 260-3) described above. For example, the wearable electronic device (101) can identify an external object included within the FoV by using a camera positioned toward the user's FoV. The identification of the external object by the wearable electronic device (101) can be performed based on a sensor for identifying the distance between the wearable electronic device (101) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV may support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable electronic device (101) may include a camera (260) (e.g., a face tracking camera) positioned toward the face to acquire an image including the face of a user wearing the wearable electronic device (101).

[0115] Although not illustrated, according to one embodiment, the wearable electronic device (101) may further include a light source (e.g., LED) that emits light toward a subject (e.g., user's eye, face, and / or an object outside the FoV) being photographed using a camera (260). The light source may include an LED of infrared wavelength. The light source may be placed in at least one of the frame (200) and hinge units (206, 207).

[0116] According to one embodiment, the battery module (270) can supply power to the electronic components of the wearable electronic device (101). In one embodiment, the battery module (270) may be placed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may each be placed in the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be placed at the end of the first temple (204) and / or the second temple (205).

[0117] According to one embodiment, the antenna module (275) can transmit a signal or power to the outside of the wearable electronic device (101) or receive a signal or power from the outside. In one embodiment, the antenna module (275) may be placed within the first temple (204) and / or the second temple (205). For example, the antenna module (275) may be placed close to one side of the first temple (204) and / or the second temple (205).

[0118] According to one embodiment, the speaker (255) can output an acoustic signal to the outside of the wearable electronic device (101, 400). The acoustic output module may be referred to as the speaker. In one embodiment, the speaker (255) may be placed within a first temple (204) and / or a second temple (205) to be placed adjacent to the ear of a user wearing the wearable electronic device (101). For example, the speaker (255) may include a second speaker (255-2) placed adjacent to the user's left ear by being placed within the first temple (204), and a first speaker (255-1) placed adjacent to the user's right ear by being placed within the second temple (205).

[0119] A light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable electronic device (101) to the user. For example, if the wearable electronic device (101) requires charging, it may emit red light at a constant frequency. In one embodiment, the light-emitting module may be placed on the first rim (201) and / or the second rim (202).

[0120] Referring to FIG. 2b, a wearable electronic device (101) according to one embodiment may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of a first temple (204) or a second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. On the PCB (290), one or more hardware components included in the wearable electronic device (101) (e.g., hardware components illustrated by different blocks in FIG. 4) may be disposed. The wearable electronic device (101) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0121] According to one embodiment, a wearable electronic device (101) may include at least one of a gyroscope sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the wearable electronic device (101) and / or the posture of a body part (e.g., head) of a user wearing the wearable electronic device (101). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyroscope sensor may be referred to as an inertial measurement unit (IMU).

[0122] According to one embodiment, a wearable electronic device (101, 400) can identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable electronic device (101) based on an IMU.

[0123] FIGS. 3a and 3b illustrate an example of the appearance of a wearable electronic device (e.g., a wearable electronic device (101)). The wearable electronic device (101) of FIGS. 3a and 3b may be an example of the wearable electronic device (101) of FIG. 1. According to one embodiment, an example of the appearance of a first surface (310) of the housing of the wearable electronic device (101) may be illustrated in FIG. 3a, and an example of the appearance of a second surface (320) opposite to the first surface (310) may be illustrated in FIG. 3b.

[0124] Referring to FIG. 3a, according to one embodiment, a first surface (310) of a wearable electronic device (101) may have a shape that is attachable to a part of a user's body (e.g., the face of the user). Although not illustrated, the wearable electronic device (101) may further include a strap for securing to a part of a user's body and / or one or more temples (e.g., a first temple (204) and / or a second temple (205) of FIG. 2a and FIG. 2b). A first display (250-1) for outputting an image to the left eye among the user's two eyes and a second display (250-2) for outputting an image to the right eye among the two eyes may be disposed on the first surface (310). The wearable electronic device (101, 400) may further include rubber or silicone packing formed on the first surface (310) to prevent interference by light different from light emitted from the first display (250-1) and the second display (250-2) (e.g., ambient light).

[0125] According to one embodiment, a wearable electronic device (101) may include cameras (260-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referenced to the eye-tracking camera (260-1) of FIG. 2B.

[0126] According to one embodiment, a wearable electronic device (101) may include cameras (260-5, 260-6) for capturing and / or recognizing a user's face. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable electronic device (101) may control an avatar representing the user in a virtual space based on motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable electronic device (101) may change the texture and / or shape of a part of the avatar (e.g., a part of the avatar representing a human face) using information obtained by the cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of the user wearing the wearable electronic device (101).

[0127] Referring to FIG. 3b, on a second surface (320) opposite to the first surface (310) of FIG. 3a, a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)), and / or a sensor (e.g., a depth sensor (330)) may be placed to acquire information related to the external environment of the wearable electronic device (101). For example, cameras (260-7, 260-8, 260-9, 260-10) may be placed on the second surface (320) to recognize external objects. The cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2b.

[0128] By using cameras (260-11, 260-12), the wearable electronic device (101) can acquire images and / or videos to be transmitted to each of the user's two eyes. Camera (260-11) may be placed on the second surface (320) of the wearable electronic device (101) to acquire an image to be displayed through a second display (250-2) corresponding to the right eye among the two eyes. Camera (260-12) may be placed on the second surface (320) of the wearable electronic device (101) to acquire an image to be displayed through a first display (250-1) corresponding to the left eye among the two eyes. Cameras (260-11, 260-12) may be referenced to the shooting camera (260-4) of FIG. 2B.

[0129] According to one embodiment, a wearable electronic device (101) may include a depth sensor (330) disposed on a second surface (320) to identify the distance between the wearable electronic device (101) and an external object. Using the depth sensor (330), the wearable electronic device (101) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable electronic device (101). Although not illustrated, a microphone may be disposed on the second surface (320) of the wearable electronic device (101, 400) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0130] Hereinafter, with reference to FIG. 4, the hardware or software configuration of the wearable electronic device (101, 400) will be described.

[0131] FIG. 4 illustrates an example of a block diagram of a wearable electronic device (101, 400) (e.g., the wearable electronic device (101) of FIG. 2a and 2b, the wearable electronic device (101) of FIG. 3a and 3b).

[0132] Referring to FIG. 4, a wearable electronic device (101, 400) according to one embodiment may include at least a portion of a processor (410) (e.g., processor (120)), a memory (415) (e.g., memory (130) of FIG. 1), a display (250) (e.g., a first display (250-1) and / or a second display (250-2) of FIG. 2a, FIG. 2b, FIG. 3a, and FIG. 3b), a sensor (420) (e.g., image sensor (421), motion sensor (422), microphone (423)), a communication circuit (430) (e.g., communication module (190) of FIG. 1), a voice output unit (425) (e.g., acoustic output module (155) of FIG. 1, and a speaker). A processor (410), memory (415), display (250), sensor (420), and / or communication circuit (430) may be electrically and / or operationally connected to one another by an electronic component such as a communication bus (402). In the present disclosure, the operational connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable electronic device (101, 400) are not limited to those shown in FIG. 4. For example, the wearable electronic device (101, 400) may include only some of the electronic components shown in FIG. 4.

[0133] A processor (410) of a wearable electronic device (101, 400) according to one embodiment may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP).

[0134] In one embodiment, the wearable electronic device (101, 400) may include one or more processors. The processor (410) may have a structure of a multi-core processor such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (410) may include a structure based on multiple core circuits (e.g., a big-little structure) distinguished by power consumption, clock, and / or computational power per unit time. In one embodiment comprising a processor (410) having a multi-core processor structure, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (410).

[0135] A memory (415) of a wearable electronic device (101, 400) according to one embodiment may include electronic components for storing data and / or instructions that are input to or output from a processor (410). The memory (415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multi-media card (eMMC). In one embodiment, the memory (415) may be referred to as storage.

[0136] In one embodiment, a display (250) of a wearable electronic device (101, 400) can output visualized information to a user of the wearable electronic device (101, 400). A display (250) arranged in front of the eyes of a user wearing the wearable electronic device (101, 400) may be placed in at least a part of the housing of the wearable electronic device (101, 400) (e.g., a first display (250-1) and / or a second display (250-2) of FIG. 2a, FIG. 2b, FIG. 3a, and FIG. 3b). For example, the display (250) may be included within a display assembly. For example, the display (250) may be controlled by a processor (410) including circuits such as a CPU (411), a GPU (graphic processing unit) (412), and / or a DPU (display processing unit) (413) to output visualized information to a user. The display (250) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (250) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). The embodiments are not limited thereto, and for example, if the wearable electronic device (101, 400) includes a lens for transmitting external light (external light or ambient light), the display (250) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (250) may be referred to as a display panel and / or a display module.The pixels included in the display (250) may be positioned toward either of the user's two eyes when worn by the user of the wearable electronic device (101, 400). For example, the display (250) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0137] In one embodiment, a sensor (420) of a wearable electronic device (101, 400) may generate electrical information that can be processed by a processor (410) and / or memory (415) from non-electronic information related to the wearable electronic device (101, 400). For example, the sensor (420) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable electronic device (101, 400). In addition to the GPS method, the sensor (420) may generate information indicating the geographic location of the wearable electronic device (101, 400) based on a global navigation satellite system (GNSS), such as Galileo or Beidou (compass). The above information may be stored in memory (415), processed by a processor (410), and / or transmitted to another electronic device distinct from the wearable electronic device (101, 400) through a communication circuit.

[0138] Referring to FIG. 4, an image sensor (421), a motion sensor (422), and a microphone (423) are shown as examples of sensors (420) included in a wearable electronic device (101, 400).

[0139] For example, the image sensor (421) may include one or more light sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate an electrical signal representing the color and / or brightness of light. The image sensor (421) may be referred to as a camera. The plurality of light sensors included in the image sensor (421) may be arranged in the form of a two-dimensional grid (4-dimensional array). The image sensor (421) may acquire the electrical signals of each of the plurality of light sensors substantially simultaneously to generate two-dimensional frame data corresponding to the light reaching the light sensors of the two-dimensional grid. For example, photo data captured using the image sensor (421) may refer to one (a) two-dimensional frame data acquired from the image sensor (421). For example, video data captured using the image sensor (421) may refer to a sequence of multiple two-dimensional frame data acquired from the image sensor (421) along a frame rate. The image sensor (421) may further include a flash light for outputting light in the direction in which the image sensor (421) receives light.

[0140] According to one embodiment, a wearable electronic device (101, 400) may include a plurality of image sensors arranged toward different directions, as an example of an image sensor (421). As described above with reference to FIGS. 2A, 2B, 3A, and 3B, the plurality of image sensors may include an eye-tracking camera (e.g., the eye-tracking camera (260-1) of FIGS. 2B and 3A) configured to be arranged toward the eyes of a user wearing the wearable electronic device (101, 400). The plurality of image sensors may include an outward camera. A processor (410) may identify the direction of the user's gaze using an image and / or video obtained from the eye-tracking camera. The eye-tracking camera may include an infrared (IR) sensor. The eye-tracking camera may be referred to as an eye sensor and / or an eye tracker.

[0141] An external camera may be positioned facing the front of a user wearing the wearable electronic device (101, 400) (e.g., a direction in which both eyes may face). The wearable electronic device (101, 400) may include a plurality of external cameras. The embodiments are not limited thereto, and the external camera may be positioned facing the external space. Using images and / or videos obtained from the external camera, the processor (410) may identify external objects. For example, the processor (410) may identify the position, shape, and / or gesture (e.g., hand gesture) of the user's hand wearing the wearable electronic device (101, 400) based on images and / or videos obtained from the external camera. Using images and / or videos of the external environment obtained from the external camera, the processor (410) may recognize or track one or more objects within the external environment.

[0142] According to one embodiment, the motion sensor (422) may output an electrical signal representing gravitational accelerations, accelerations, and / or angular velocities of a plurality of axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other and are based on a designated origin within the wearable electronic device (101, 400) and / or the motion sensor (422). For example, the processor (410) may repeatedly receive or acquire sensor data from the motion sensor (422), including accelerations, angular velocities, and / or magnitudes of the magnetic field of a number of the plurality of axes, based on a designated period (e.g., 1 millisecond). In one embodiment, the motion sensor (422) may be referred to as an inertial measurement unit (IMU). The sensor (420) included in the wearable electronic device (101, 400) is not limited to the above and may include a grip sensor, a proximity sensor, a heart rate sensor, a fingerprint sensor, an ambient light sensor and / or a ToF sensor. Using the motion sensor (422), the processor (410) can detect motion of the wearable electronic device (101, 400) (e.g., motion of the wearable electronic device (101, 400) caused by a user wearing the wearable electronic device (101, 400).

[0143] According to one embodiment, the microphone (423) can convert sound into an electrical signal and provide the converted electrical signal to the processor (410). The microphone (423) can be used to convert the user's voice into an electrical signal and provide it to the processor (410). Through this, the wearable electronic device (101, 400) can perform operations according to the user's voice command.

[0144] In one embodiment, a communication circuit (430) of a wearable electronic device (101, 400) may include a hardware component for supporting the transmission and / or reception of a signal between the wearable electronic device (101, 400) and an external electronic device (e.g., electronic device (102), electronic device (104)). The communication circuit (430) may include, for example, at least one of a modem, an antenna, and an O / E (optic / electronic) converter. The communication circuit (430) may support the transmission and / or reception of an electrical signal based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), WiFi (wireless fidelity), Bluetooth, BLE (Bluetooth low energy), Zigbee, LTE (long term evolution), and 5G NR (new radio).

[0145] In one embodiment, within the memory (415) of a wearable electronic device (101, 400), one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by a processor (410) of the wearable electronic device (101, 400) may be stored. A set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, sub-routine, and / or software application (hereinafter referred to as an application). For example, the wearable electronic device (101, 400), and / or processor (410) may perform at least one of the operations described below when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, program, and / or software application is executed. In the following, the statement that a software application is installed within a wearable electronic device (101, 400) means that one or more instructions provided in the form of a software application (or package) are stored in memory (415), and that said one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the wearable electronic device (101, 400)) by the processor (410). For example, the application may include a program and / or library related to a service provided to a user.

[0146] Referring to FIG. 4, programs installed on a wearable electronic device (101, 400) may be included in any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, within the hardware abstraction layer (480), programs (e.g., modules, or drivers) designed to target the hardware of the wearable electronic device (101, 400) (e.g., a display (250), and / or a sensor (420)) may be included. The framework layer (450) may be referred to as an XR framework layer in that it includes one or more programs for providing XR (extended reality) services. For example, the layers illustrated in FIG. 4 are logically (or for convenience of explanation) separated, and may not imply that the address space of memory (415) is separated by said layers.

[0147] According to one embodiment, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or application layer (440) (e.g., location tracker (471), spatial recognizer (472), gesture tracker (473), and / or eye tracker (474)) may be included. The programs included in the framework layer (450) may provide an application programming interface (API) that is executable (or invokeable) based on other programs.

[0148] According to one embodiment, the application layer (440) may include a program designed to target a user of a wearable electronic device (101, 400). Examples of programs included in the application layer (440) include an XR (extended reality) system UI (user interface) (441) and / or an XR application (442), but the embodiment is not limited thereto. For example, programs included in the application layer (440) (e.g., software applications) may call an API to cause the execution of a function supported by programs included in the framework layer (450).

[0149] According to one embodiment, a wearable electronic device (101, 400) may display one or more visual objects on a display (250) to perform interaction with a user based on the execution of an XR system UI (441). A visual object may mean an object that can be placed on a screen for the transmission of information and / or interaction, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable electronic device (101, 400) may provide the user with functions available in a virtual space based on the execution of an XR system UI (441).

[0150] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plugin (444) are depicted within the XR system UI (441), but are not limited thereto. For example, based on the XR system UI (441), the processor (410) may execute a lightweight renderer (443) and / or an XR plugin (444) within the framework layer (450).

[0151] According to one embodiment, a wearable electronic device (101, 400) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute a rendering pipeline that is partially modified, based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that is partially modified. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, the wearable electronic device (101, 400) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute the entire rendering pipeline based on the execution of an XR plugin (444). The XR plugin (444) can be referred to as an open XR native client in terms of defining (or setting) the entire rendering pipeline.

[0152] According to one embodiment, a wearable electronic device (101, 400) may display a screen representing at least a portion of a virtual space on a display (250) based on the execution of an XR application (442). An XR plugin (444-1) included in the XR application (442) may include instructions that support functions similar to the XR plugin (444) of the XR system UI (441). Descriptions of the XR plugin (444-1) that overlap with descriptions of the XR plugin (444) may be omitted. The wearable electronic device (101, 400) may trigger the execution of a virtual space manager (451) based on the execution of the XR application (442).

[0153] According to one embodiment, a wearable electronic device (101, 400) may display an image on a display (250) in a virtual space based on the execution of an application (445). The application (445) may be configured to output image information for displaying a two-dimensional image. The wearable electronic device (101, 400) may trigger the execution of a virtual space manager (451) based on the execution of the application (445). The wearable electronic device (101, 400) may generate dual image information to represent the two-dimensional image in a three-dimensional virtual space based on the execution of the application (445). Here, the dual image information may include a first image information for the left eye and a second image information for the right eye, taking into account binocular parallax. In order to display the above two-dimensional image in a three-dimensional virtual space, the wearable electronic device (101, 400) can generate the dual image information based on image information for displaying the above two-dimensional image.

[0154] According to one embodiment, a wearable electronic device (101, 400) may provide a virtual space service based on the execution of a virtual space manager (451). For example, the virtual space manager (451) may include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (451), the wearable electronic device (101, 400) may identify a virtual space formed based on the user's location indicated by data acquired through a sensor (420), and may display at least a portion of the virtual space on a display (250). The virtual space manager (451) may be referred to as a composition presentation manager (CPM).

[0155] According to one embodiment, the virtual space manager (451) may include a runtime service (452). For example, the runtime service (452) may be referred to as an OpenXR runtime module (or OpenXR runtime program). A wearable electronic device (101, 400) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (452). For example, the wearable electronic device (101, 400) may perform rendering for a virtual space service for the user based on the execution of the runtime service (452). For example, a virtual space-related function executable by the application layer (440) may be supported based on the execution of the runtime service (452).

[0156] According to one embodiment, the virtual space manager (451) may include a pass-through manager (453). The wearable electronic device (101, 400) may display an image and / or video representing a real space acquired through an external camera superimposed on at least a portion of the screen while displaying a screen representing a virtual space on the display (250) based on the execution of the pass-through manager (453).

[0157] According to one embodiment, the virtual space manager (451) may include an input manager (454). The wearable electronic device (101, 400) may identify acquired data (e.g., sensor data) by executing one or more programs included within the recognition service layer (470) based on the execution of the input manager (454). The wearable electronic device (101, 400) may identify user inputs associated with the wearable electronic device (101, 400) using the acquired data. The user inputs may be associated with user motions (e.g., hand gestures), gaze, and / or speech identified by a sensor (420) (e.g., an image sensor (421) such as an external camera). The user inputs may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0158] According to one embodiment, a perception abstract layer (460) may be used for data exchange between a virtual space manager (451) and a perception service layer (470). In terms of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) may be referred to as an interface. For example, the perception abstract layer (460) may be referred to as OpenPX. The perception abstract layer (460) may be used for a perception client and a perception service.

[0159] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data obtained from the sensor (420). The one or more programs may include at least one of a location tracker (471), a spatial recognizer (472), a gesture tracker (473), and / or an eye tracker (474). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those shown in FIG. 4.

[0160] According to one embodiment, the wearable electronic device (101, 400) can identify the posture of the wearable electronic device (101, 400) using a sensor (420) based on the execution of a position tracker (471). The wearable electronic device (101, 400) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable electronic device (101, 400) using data acquired using an external camera (e.g., image sensor (421)) and / or an IMU (e.g., motion sensor (422) including a gyroscope, accelerometer, and / or geomagnetic sensor) based on the execution of the position tracker (471). The position tracker (471) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0161] According to one embodiment, a wearable electronic device (101, 400) can obtain information to provide a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable electronic device (101, 400) (or a user of the wearable electronic device (101, 400)) based on the execution of a spatial recognizer (472). Based on the execution of the spatial recognizer (472), the wearable electronic device (101, 400) can reproduce the surrounding environment of the wearable electronic device (101, 400) in three dimensions using data obtained using an external camera (e.g., an image sensor (421)). A wearable electronic device (101, 400) can identify at least one of a plane, an incline, and a staircase based on the surrounding environment of the wearable electronic device (101, 400) reproduced in three dimensions based on the execution of a spatial recognizer (472). The spatial recognizer (472) may be referred to as a scene understanding (SU) module (or scene understanding program).

[0162] According to one embodiment, the wearable electronic device (101, 400) can identify (or recognize) the pose and / or gesture of the user's hand of the wearable electronic device (101, 400) based on the execution of the gesture tracker (473). For example, the wearable electronic device (101, 400) can identify the pose and / or gesture of the user's hand using data acquired from an external camera (e.g., image sensor (421)) based on the execution of the gesture tracker (473). For example, the wearable electronic device (101, 400) can identify the pose and / or gesture of the user's hand based on data (or images) acquired using an external camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0163] According to one embodiment, the wearable electronic device (101, 400) can identify (or track) the movement of the user's eyes of the wearable electronic device (101, 400) based on the execution of the eye tracker (474). For example, the wearable electronic device (101, 400) can identify the movement of the user's eyes using data obtained from an eye-tracking camera (e.g., image sensor (421)) based on the execution of the eye tracker (474). The eye tracker (474) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0164] According to one embodiment, the recognition service layer (470) of the wearable electronic device (101, 400) may further include a face tracker (475) for tracking the user's face. For example, the wearable electronic device (101, 400) may identify (or track) the movement of the user's face and / or the user's facial expression based on the execution of the face tracker (475). The wearable electronic device (101, 400) may estimate the user's facial expression based on the movement of the user's face based on the execution of the face tracker (475). For example, the wearable electronic device (101, 400) may identify the movement of the user's face and / or the user's facial expression based on data (e.g., images and / or videos) obtained using a camera (425) (e.g., a camera facing at least a part of the user's face) based on the execution of the face tracker (475).

[0165] Referring to FIG. 4, an example of a processor (410) is shown as a CPU (411), a GPU (graphic processing unit) (412), and / or a DPU (display processing unit) (413). A renderer (490) may include instructions for rendering images in a three-dimensional virtual space. A processor (410) (e.g., DPU (413)) that executes the renderer (490) may acquire at least one image to be displayed at least partially in a display area of ​​a display (250) in a software application (e.g., a software application executed by the CPU (411) and / or GPU (412)). For example, a processor (410) that executes the renderer (490) may determine the location of an area where an application (e.g., an XR application (442), an application (445)) will be rendered. The processor (410) that executed the renderer (490) can generate an image of the application to be displayed on the display (250). The renderer (490) can synthesize images to generate a composite image to be displayed on the display (250).

[0166] According to one embodiment, a processor (410) that executes a renderer (490) can divide the display area of ​​a display (250) into a foveated portion (or may be referred to as a foveated area) and a peripheral portion (or may be referred to as a residual area) using a gaze position calculated using a position tracker (471) and / or a gaze tracker (474). For example, a processor (410) that detects coordinate values ​​of a gaze position can determine the portion of the display area containing said coordinate values ​​as the foveated area. A DPU (413) that executes a renderer (490) can acquire at least one image corresponding to each of said foveated area and said residual area, having a size smaller than the size of the entire display area of ​​the display (250) or having a resolution less than the resolution of the display area.

[0167] According to one embodiment, a processor (410) that executes a renderer (490) can obtain or generate a composite image to be displayed on a display (250) by synthesizing an image corresponding to a foveated area and an image corresponding to a surrounding area. For example, the processor (410) can perform upscaling to enlarge the image corresponding to the surrounding area to the size of the entire display area of ​​the display (250). On the enlarged image, the processor (410) can combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (250). Along the boundary line of the image corresponding to the foveated area, the processor (410) can mix the enlarged image and the image corresponding to the foveated area by applying a visual effect such as blur.

[0168] FIG. 5 illustrates an example of a block diagram of an electronic device (e.g., electronic device (101), wearable electronic device (101, 400)) for displaying an image in a virtual space. FIG. 5 describes an example in which multiple programs / instructions for displaying an image in a virtual space are executed. The multiple programs / instructions may all be executed on a single processor (e.g., AP) or may be executed by multiple processors (e.g., AP, GPU (graphic processing unit), NPU (neural processing unit)). The meaning of being able to be executed by multiple processors is that some programs / instructions may be executed by a first processor and other programs / instructions may be executed by a second processor different from the first processor.

[0169] The wearable electronic device of FIG. 5 (e.g., the electronic device (101) of FIG. 1, the wearable electronic device of FIG. 2a to 3b, the wearable electronic device (400) of FIG. 4) may execute a virtual space manager (550) (e.g., the virtual space manager (451) of FIG. 4, CPM) to render an image in a virtual space. For the virtual space manager (550), at least some of the descriptions of the virtual space manager (451) of FIG. 4 may be referenced. The virtual space manager (550) may include a platform for supporting virtual space services. The virtual space manager (550) may include a runtime service (551) (e.g., OpenXR Runtime), a panel rendering (552) (e.g., 2D Panel Render), and an XR composite unit (553) (XR Compositor). A wearable electronic device (101, 400) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of a runtime service (551). For the runtime service (551), at least some of the descriptions of the runtime service (452) of FIG. 4 may be referenced. A wearable electronic device (101, 400) may display at least one image (video) on a panel (e.g., a 2D panel) to enable the implementation of a virtual space through a display based on the execution of panel rendering (552). For example, a wearable electronic device (101, 400) may display a rendering image corresponding to RGB information (566) for a panel from a spatialization manager (540) described later through a display (e.g., a display (250)). A wearable electronic device (101, 400) can synthesize an image of a real area (hereinafter, pass-through image) and a virtual area image captured through a camera in a virtual space based on the execution of an XR composite unit (553) (XR Compositor).For example, a wearable electronic device (101, 400) can generate a composite image by merging the pass-through image and the virtual region image based on the execution of the XR composite unit (553). The wearable electronic device (101, 400) can transmit the generated composite image to a display buffer so that the composite image is displayed. The wearable electronic device (101, 400) can identify a virtual space through a virtual space manager (550) and display at least a portion of the virtual space on a display (250). The virtual space manager (550) may be referred to as CPM. The electronic device (101) can execute the virtual space manager (550) to render an image corresponding to at least a portion of the virtual space.

[0170] According to one embodiment, a wearable electronic device (101, 400) may execute a spatialization manager (540). The spatialization manager (540) may perform processes for displaying an image in a three-dimensional virtual space. The wearable electronic device (101, 400) may perform preprocessing based on the execution of the spatialization manager (540) so that an image can be rendered in a three-dimensional virtual space through a virtual space manager (550). For example, the wearable electronic device (101, 400) may perform at least some of the functions of the renderer (490) of FIG. 4 based on the execution of the spatialization manager (540). A wearable electronic device (101, 400) can process image information provided by an application (e.g., an XR application (510), an application providing a non-XR general 2D screen (520), or an application providing a system UI (530)) based on the execution of a spatialization manager (540). The spatialization manager (540) (e.g., Space Flinger) may include a system screen manager (541) (e.g., System scene), an input manager (542) (e.g., Input Routing), and a lightweight rendering engine (543) (e.g., Impress Engine). The system screen manager (541) may be executed to display a system UI (530). System UI-related information (564) may be transmitted to the system screen manager (541) from a program (e.g., API) that provides the system UI (530). System UI-related information (564) may be obtained through a spatializer API and / or a Same-process private API. The spatialization manager (540) can determine the layout (e.g., position, display order) of the screen of the system UI (530) in three dimensions through pre-allocated resources.The system screen manager (541) may transmit image information (567) to the virtual space manager (550) for rendering the screen of the system UI (530) according to the layout. The input manager (542) may be configured to process user input (e.g., user input on the system screen or app screen). The lightweight rendering engine (543) may be a renderer for image generation (e.g., lightweight renderer (443)). For example, the lightweight rendering engine (543) may be used to display the system UI (530). According to one embodiment, the spatialization manager (540) may include a lightweight rendering engine (543) for rendering the system UI. According to one embodiment, if the lightweight rendering engine (543) does not have sufficient resources to render an avatar used in an HMD, at least one external rendering engine may be used. At this time, to resolve compatibility issues with external rendering (e.g., 3rd party engine), an external rendering engine support module may be added inside the spatialization manager (540).

[0171] According to one embodiment, the electronic device may execute an application. For example, in response to the execution of an XR application (510) (e.g., XR application (442), 3D game, XR map, other immersive application), the virtual space manager (550) may be executed. The wearable electronic device (101, 400) may provide dual image information (561) provided from the XR application (510) to the virtual space manager (550). To display images in three-dimensional space, the dual image information (561) may include two image information that takes into account binocular parallax. For example, the dual image information (561) may include a first image information for the user's left eye and a second image information for the user's right eye to render in three-dimensional virtual space. Hereinafter, the term dual image information is used in the present disclosure as a term referring to image information for displaying images for both eyes in three-dimensional space. In addition to the dual image information, the above dual image information may utilize binocular image information, dual image information, dual image data, dual image, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimension conversion image data, binocular parallax image data, and / or equivalent technical terms. The electronic device (101) can generate a composite image by merging image layers through a virtual space manager (550). The wearable electronic device (101, 400) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (250) of the wearable electronic device (101, 400).

[0172] According to one embodiment, the electronic device may execute at least one application among an XR application (510) and other applications (520) (e.g., a first application (520-1), a second application (520-2), …, an Nth application (520-N)). According to one embodiment, the application (520) may be configured to output image information for displaying a two-dimensional image. In other words, the application (520) may provide a two-dimensional image. For example, the application (520) may be a video application, a schedule application, or an internet browser application. Let us assume that, in response to the execution of the application (520), image information (562) provided by the application (520) is provided to the virtual space manager (550). Since the image information (562) has only x-coordinates and y-coordinates within a two-dimensional plane, it may be difficult to consider the sequential relationship between other applications centered on the user (i.e., distance from the user). The electronic device (101) may execute a spatialization manager (540) to provide dual image information to a virtual space manager (550), even when displaying an application (520) that provides a general 2D screen. For example, based on the execution of the spatialization manager (540), the electronic device (101) may receive application-related information (563) from a first application (520-1). For example, the application-related information (563) may include image information representing a 2D image of the first application (520-1) (e.g., information including RGB per pixel) and / or content information in the first application (520-1) (e.g., characteristics of the content executed in the first application, type of content). The application-related information (563) may be obtained through a spatializer API.Based on the execution of the spatialization manager (540), the electronic device (101) can identify information (hereinafter, location information) regarding the location of the area to be rendered and the size of the area to be rendered. Based on the execution of the spatialization manager (540), the wearable electronic device (101, 400) can generate dual image information (565, e.g., RGBx2) that takes into account the user's binocular parallax through the image information and the location information. Based on the execution of the spatialization manager (540), the electronic device (101) can provide the dual image information (565) to the virtual space manager (550). By converting a simple two-dimensional image into dual image information (565), the problem caused by the image information (562) being directly transmitted to the virtual space manager (550) can be resolved. Additionally, as at least some of the functions for displaying images in virtual space are performed by the spatialization manager (540) instead of the virtual space manager (550), the burden on the virtual space manager (550) may be reduced. However, as image information from the application (520) is not transmitted directly to the virtual space manager (550) but is transmitted through the spatialization manager (540), the quality of the image ultimately output to the user may be lowered. For example, in the first application (520-1), an image is rendered at a resolution of approximately 2756 x 1846, but the image may be downsampled during the process of being transmitted to the virtual space manager (550) through the spatialization manager (540) (e.g., downsampled from a resolution of approximately 2756 x 1846 to a resolution of approximately 1160 x 680). Afterwards, the virtual space manager (550) can upsample the downsampled image (e.g., upsample from a resolution of about 1160 x 680 to a resolution of about 1625 x 1070) and pass the upsampled image to the display buffer.As such, in the process of an image being transmitted from an application (520) to a spatialization manager (540) and from a spatialization manager (540) to a virtual space manager (550), a resolution mismatch may occur, or an aliasing problem or a degradation of image quality may occur during the upsampling process. To resolve the above-mentioned problems, the present disclosure describes techniques for controlling the resolution of an area to be displayed in an application and performing foveation rendering based on the system structure illustrated in FIG. 5.

[0173] Figure 6 illustrates an example of a structure of multiple layers.

[0174] Referring to FIG. 6, programs installed in a wearable electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device of FIG. 2a to 3b, the wearable electronic device (400) of FIG. 4) can be classified into one of a platform layer (610), a recognition service layer (620) (e.g., the recognition service layer (470) of FIG. 4), and a sensor service layer (630). For example, the wearable electronic device (101, 400) can operate based on the platform layer (610), the recognition service layer (620), and the sensor service layer (630).

[0175] According to one embodiment, the platform layer (610) may be configured for XR services. For example, the platform layer (610) may include a platform (e.g., an Android platform) for supporting XR services. For example, the platform layer (610) may include the virtual space manager (550) of FIG. 5. The platform layer (610) may include a runtime service (611). For the runtime service (611), the descriptions of the runtime service (551) of FIG. 5 and the descriptions of the runtime service (452) of FIG. 4 may be referenced. For example, the runtime service (611) may be referenced as an OpenXR runtime module. The runtime service (611) may be used to provide at least one of a user pose prediction function, a frame timing function, and / or a spatial input function through a wearable electronic device (101, 400). For example, the runtime service (611) may be used to perform rendering for XR services for the user. For example, based on the runtime service (611), an application (e.g., a Unity or OpenXR native application) can be implemented.

[0176] According to one embodiment, the perception abstraction layer (612) may be used for data exchange between the platform layer (610) and the perception service layer (620). For the perception abstraction layer (612), the descriptions of the perception abstraction layer (460) of FIG. 4 may be referenced. For example, the perception abstraction layer (612) may be referenced as OpenPX. The perception abstraction layer (612) may be used for a perception client and a perception service.

[0177] According to one embodiment, the recognition service layer (620) may include a service module (621), a recognition plugin layer (622), a sensor management module (623), a playback module (624), and / or an external data management module (625). For example, the recognition service layer (620) may include at least one of the service module (621), the recognition plugin layer (622), the sensor management module (623), the playback module (624), and / or an external data management module (625). For example, at least some of the service module (621), the recognition plugin layer (622), the sensor management module (623), the playback module (624), and the external data management module (625) may be omitted.

[0178] According to one embodiment, the service module (621) can manage input data of the wearable electronic device (101, 400). The service module (621) can be used to manage data (e.g., gesture information) obtained from a plurality of recognition modules included in the recognition plugin layer (622). As an example, the service module (621) may be referred to as SxrDataService.

[0179] According to one embodiment, the service module (621) can perform interfacing with an upper layer (e.g., platform layer (610) or runtime service (611)). The service module (621) can exchange data with the upper layer (e.g., platform layer (610) or runtime service (611)) through the recognition abstraction layer (612). For example, the recognition abstraction layer (612) may be referred to as OpenPX. According to an embodiment, the service module (621) may support OpenXR Extension as well as OpenPX. The service module (621) may be used to exchange data (e.g., gesture information) between a plurality of recognition modules. The service module (621) may be configured to manage data processed in the recognition service layer (620). The service module (621) may select data among the data to be recognized as input to the wearable electronic device (101, 400). The above data may include data obtained from a plurality of recognition modules and data obtained through an external data management module (625). The service module (621) may manage data to be used in the recognition abstraction layer (612). The service module (621) may select data to be recognized as an input to the wearable electronic device (101, 400) from the above data and provide it to the recognition abstraction layer (612).

[0180] According to one embodiment, the recognition plugin layer (622) may include a plurality of recognition modules. The plurality of recognition modules may be referred to as a plurality of perception solutions.

[0181] For example, a plurality of recognition modules may include at least one of a head tracking (HeT) module (622-1), a scene understanding (SU) module (622-2), a hand tracking (HaT) module (622-3), an eye tracking (ET) module (622-4), and a face tracking (FT) module (622-5). Each of the plurality of recognition modules included in the recognition plugin layer (622) may include a common interface for connection (or interaction) with a sensor management module (623). Each of the plurality of recognition modules may include a common interface for connection (or interaction) with a sensor management module (623).

[0182] According to one embodiment, the head tracking module (622-1) can identify the posture and orientation of the wearable electronic device (101, 400) using at least one sensor of the wearable electronic device (101, 400). For example, the head tracking module (622-) can identify the 6 degrees of freedom pose (6 dof pose) and orientation of the wearable electronic device (101, 400) based on data acquired using a camera (e.g., image sensor (421) of FIG. 4) and an IMU.

[0183] According to one embodiment, the environment recognition module (622-2) may be used to construct the surrounding environment of a wearable electronic device (101, 400) (or a user of the wearable electronic device (101, 400)) into a three-dimensional virtual space. The environment recognition module (622-2) may be used to reconstruct the surrounding environment of the wearable electronic device (101, 400) in three dimensions based on data acquired using a camera (e.g., the image sensor (421) of FIG. 4). Based on the surrounding environment of the wearable electronic device (101, 400) reconstructed in three dimensions, the environment recognition module (622-2) may identify at least one of a plane, an incline, and a staircase.

[0184] According to one embodiment, a hand tracking module (622-3) may be used to identify (or recognize) the pose and / or gesture of a user's hand of a wearable electronic device (101, 400). For example, the hand tracking module (622-3) may identify the pose and / or gesture of the user's hand based on data acquired from at least one sensor. For example, the hand tracking module (622-3) may identify the pose and / or gesture of the user's hand based on data (e.g., an image) acquired using a camera.

[0185] According to one embodiment, an eye tracking module (622-4) may be used to identify (or track) the eye movements of a user of a wearable electronic device (101, 400). For example, the eye tracking module (622-4) may identify the user's eye movements based on data obtained from at least one sensor. For example, the eye tracking module (622-4) may identify the user's eye movements based on data obtained using a camera (e.g., the eye tracking camera (260-1) of FIG. 2b and FIG. 3a)) and / or an IR LED (infrared light emitting diode). For example, the eye tracking module (622-4) may be used to obtain the direction of gaze and the position of gaze of a user wearing the wearable electronic device (101, 400).

[0186] According to one embodiment, the face tracking module (622-5) may be used to identify (or track) the movement of the user's face and / or the user's facial expression. The face tracking module (622-5) may estimate the user's facial expression based on the movement of the user's face. For example, the face tracking module (622-5) may identify the movement of the user's face and / or the user's facial expression based on data (e.g., images) obtained using a camera (e.g., camera (260) of FIG. 2a and FIG. 2b).

[0187] For example, a plurality of recognition modules included in the recognition plugin layer (622) may be configured in a plugin structure. For example, some of the plurality of recognition modules may be replaced with other modules regardless of the sensor service layer (630) and platform layer (610), which are lower layers of the recognition service layer (620).

[0188] According to one embodiment, the sensor management module (623) may be used to provide (or transmit) data to each of the plurality of recognition modules through a common interface. For example, the sensor management module (623) may be used to separate (or eliminate) the dependency between the lower layer, the sensor service layer (630), and the upper layer, the recognition plugin layer (622). For example, the sensor management module (623) may be referred to as SxrSensorSeviceManger.

[0189] According to one embodiment, the sensor management module (623) may support various modules (or sensor services) of the sensor service layer (630). Multiple recognition modules may not directly interface with the sensor service layer (630). Multiple recognition modules may receive data (e.g., sensor data) through the sensor management module (623). Therefore, even if a module of the sensor service layer (630) is changed, it may not affect the multiple recognition modules.

[0190] According to one embodiment, the sensor management module (623) may further include a load balancing module. The load balancing module may identify data provided from the sensor service layer (630). Based on the data provided from the sensor service layer (630), the load balancing module may identify at least some of the recognition modules among the plurality of recognition modules. The load balancing module may provide data to the identified at least some of the recognition modules. For example, the load balancing module may distribute data to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable electronic device (101, 400). For example, the load balancing module may filter the data provided to the plurality of recognition modules based on the status of the plurality of recognition modules and / or the status of the wearable electronic device (101, 400). According to an embodiment, the load balancing module may be configured independently of the sensor management module (623). The load balancing module can be referenced as SxrPerceptionLoadBalancer.

[0191] According to one embodiment, a playback module (624) may be used to provide a stored dataset to at least one of a plurality of recognition modules in real time via playback. For example, the dataset may be stored through the playback module (624) based on specified specifications. The dataset may include first data obtained from the sensor service layer (630) as well as second data obtained based on the first data obtained from the sensor service layer (630) (e.g., virtual object data or synthetic data). For example, the first data may be referenced as sensor data. The second data may be referenced as virtual data.

[0192] According to an embodiment, the wearable electronic device (101, 400) may receive data from an external electronic device. For example, the data received from the external electronic device may include first data obtained from a service layer included in the external electronic device and / or second data obtained based on the first data. The wearable electronic device (101, 400) may perform playback (or a playback function) using the data received from the external electronic device. The wearable electronic device (101, 400) may transmit the result of the playback (or playback function) to the external electronic device. For example, the wearable electronic device (101, 400) may be used to process the data obtained from the external electronic device on its behalf. The wearable electronic device (101, 400) may receive data obtained from at least one sensor of the external electronic device. A wearable electronic device (101, 400) can acquire information (e.g., information about 6 degrees of freedom posture) through a playback module (624) (or a plurality of recognition modules) based on received data. The wearable electronic device (101, 400) can transmit the acquired information to an external electronic device. The external electronic device can provide XR services based on the acquired information.

[0193] According to one embodiment, the playback module (624) can perform playback (or a playback function) based on at least one of the first data and the second data. According to an embodiment, the playback module (624) can perform playback by combining (or mixing) real-time data (e.g., runtime data) and pre-stored data.

[0194] For example, playback may refer to a function that utilizes stored data (or gesture information) according to the operation of a wearable electronic device (101, 400). For example, playback may refer to a function that identifies a value regarding the performance of an XR service through a comparison between gesture information obtained based on a specified operation regarding an XR service and reference gesture information according to said specified operation.

[0195] For example, playback may refer to a function for obtaining information on the performance of an XR service provided to a user of a wearable electronic device (101, 400). The playback module (624) may identify information (e.g., gesture information) regarding a user who has performed a designated action (e.g., mission) regarding the XR service. The playback module (624) may identify reference information regarding the designated action. Reference information may refer to information for determining the completion of the performance of the designated action. The playback module (624) may identify the similarity between the information regarding the user who has performed the designated action and the reference information. Based on the similarity, the playback module (624) may identify whether the performance of the action designated by the user has been completed.

[0196] According to an embodiment, the playback module (624) may be included in the sensor management module (623). For example, the playback module (624) can perform playback through the sensor management module (623) without changing the plurality of recognition modules.

[0197] According to one embodiment, an external data management module (625) may be used to manage data obtained through an external electronic device (e.g., a smart watch, a smartphone, or a tablet PC) (or at least one sensor of the external electronic device) connected to a wearable electronic device (101, 400). For example, the external data management module (625) may improve the accuracy of a plurality of recognition modules using data obtained from an external electronic device. For example, the external data management module (625) may correct data (or gesture information) obtained from a plurality of recognition modules using data obtained from an external electronic device. According to an embodiment, the external data management module (625) may not be included in the recognition service layer (620).

[0198] According to one embodiment, the sensor service layer (630) may be used to control at least one sensor (e.g., camera, IMU, TOF (time of flight) sensor). For example, the sensor service layer (630) may be used to provide a service for accessing at least one sensor. For example, the sensor service layer (630) may include at least one of a module for VR services (e.g., QVRservice), a module for XR services (e.g., SxrSensorService), a sensor API (e.g., android sensor API), and a sensor hardware abstraction layer (sensor HAL).

[0199] According to one embodiment, the sensor management module (623) can provide sensor data to the recognition plugin layer (622) through a common interface. For example, the sensor management module (623) can provide sensor data to each of a plurality of recognition modules through the same interface. For example, the sensor management module (623) can provide sensor data according to the operation of the recognition module to the recognition module without changing the configuration information of the recognition plugin layer (622) based on changing (or modifying) the configuration information (e.g., configuration file) regarding the sensor management module (623).

[0200] According to one embodiment, the sensor management module (623) can identify sensor data for at least one recognition module based on the operation of at least one recognition module among a plurality of recognition modules. The sensor management module (623) can provide the identified sensor data to the at least one recognition module.

[0201] According to one embodiment, when the head tracking module (622-1) is driven, the sensor management module (623) can acquire camera data and IMU data through at least one of a module for VR services, a module for XR services, a sensor API, and a sensor hardware abstraction layer in the sensor service layer (630). The sensor management module (623) can provide the camera data and IMU data to the head tracking module (622-1). According to an embodiment, the camera data and IMU data may be acquired through different modules.

[0202] According to one embodiment, when the environment recognition module (622-2) is operated in playback mode, the sensor management module (623) can identify stored camera data and stored posture data. The sensor management module (623) can provide the camera data and posture data to the environment recognition module (622-2).

[0203] According to one embodiment, the service module (621) may be configured to eliminate dependency on the upper layer of the recognition plugin layer (622). For example, the upper layer of the recognition plugin layer (622) may include a platform layer (610) (e.g., Android XR) and / or an application layer (e.g., the application layer (440) of FIG. 4).

[0204] According to one embodiment, a service module (621) can manage input data of a wearable electronic device (101, 400). The service module (621) may be configured to integrate and manage information (e.g., gesture information or tracking data) obtained from a plurality of recognition modules. The service module (621) may convert the information (e.g., gesture information or tracking data) according to the requirements of a higher layer without changing the plurality of recognition modules, and then provide the converted information to the higher layer.

[0205] For example, a service module (621) can obtain information about a 6-degree-of-freedom attitude from a head tracking module (622-1). The information about the 6-degree-of-freedom attitude obtained from the head tracking module (622-1) can be configured in a quaternion format. On the other hand, an upper layer (e.g., platform layer (610)) can request information about the 6-degree-of-freedom attitude configured in an axis-angle representation format. The service module (621) can change (or convert) the information about the 6-degree-of-freedom attitude configured in a quaternion format into information about the 6-degree-of-freedom attitude configured in an axis-angle representation format. The service module (621) can provide information about the 6-degree-of-freedom attitude configured in an axis-angle representation format to the upper layer (e.g., platform layer (610)). However, it is not limited thereto. For example, the service module (621) can change (or convert) information about a 6-degree-of-freedom attitude configured in an axis-angle representation format into information about a 6-degree-of-freedom attitude configured in a quaternion format and provide it to the upper layer.

[0206] For example, a service module (621) can obtain information about the movement of the hand from a hand tracking module (622-3). Information about the movement of the hand can be obtained based on the movement of a first number of joints. On the other hand, an upper layer (e.g., platform layer (610)) can request information about the movement of the hand obtained based on the movement of a second number of joints. The service module (621) can perform either a joint interpolation procedure or a simplification procedure. Based on performing either a joint interpolation procedure or a simplification procedure, the service module (621) can support the structure of the joints required by the upper layer.

[0207] FIG. 7a is a block diagram of a wearable electronic device according to an embodiment of the present disclosure.

[0208] In describing the configurations of the wearable electronic device (400) shown in FIG. 7a, the description of configurations identical to those of the wearable electronic device (400) shown in FIG. 4 may be omitted.

[0209] Referring to FIG. 7a, a wearable electronic device (400) according to one embodiment of the present disclosure (e.g., electronic device (101) of FIG. 1, wearable electronic device of FIG. 2a to 3b, wearable electronic device (400) of FIG. 4) may include a display (250) (e.g., display (250) of FIG. 4), a processor (410) (e.g., processor (120) of FIG. 1, processor (410) of FIG. 4), a sensor (420) (e.g., sensor (420) of FIG. 4), a voice output unit (425) (e.g., voice output unit (425) of FIG. 4), a user input unit (701), and an artificial intelligence model (702).

[0210] According to one embodiment, the user input unit (701) may include the recognition service layer (470) of FIG. 4.

[0211] For example, the user input unit (701) can track the location of a user wearing a wearable electronic device (101, 400). For example, the user input unit (701) can use a GPS sensor to locate the geographic location of a user wearing a wearable electronic device (101, 400) and generate location tracking information for the user. The user input unit (701) can provide the location tracking information for the user to a processor (410) or a notification management unit (414).

[0212] For example, the user input unit (701) can recognize the space where a user wearing a wearable electronic device (101, 400) is located. The user input unit (701) can obtain information for providing a three-dimensional virtual space corresponding to the user's surrounding environment (e.g., external space). The user input unit (701) can provide spatial recognition information of the user's location to the processor (410) or the notification management unit (414).

[0213] For example, the user input unit (701) can track the gestures of a user wearing a wearable electronic device (101, 400). The user input unit (701) can provide the user's gesture tracking information to a processor (410) or a notification management unit (414).

[0214] For example, the user input unit (701) can track the gaze of a user wearing a wearable electronic device (101, 400). The user input unit (701) can identify the gaze of the user (e.g., the movement of the user's eyes) by using data obtained from a gaze tracking camera (e.g., camera) based on the execution of a gaze tracker (e.g., the gaze tracker (474) of FIG. 4). The user input unit (701) may include an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module. The user input unit (701) can provide the user's gaze tracking information to a processor (410) or a notification management unit (414).

[0215] According to one embodiment, the processor (410) may include a notification management unit (414).

[0216] For example, the notification management unit (414) can determine whether a user wearing a wearable electronic device (101, 400) is located between multiple virtual objects. Since a notification is required when the user is located between multiple virtual objects, the notification management unit (414) can provide monitoring information regarding the user being located between multiple virtual objects to the artificial intelligence model (702).

[0217] For example, the notification management unit (414) can determine whether a user wearing a wearable electronic device (101, 400) and a virtual object are located close to each other within a set distance (e.g., reference distance). Since a notification is required when the distance between the user and the virtual object is close within a set distance, the notification management unit (414) can provide monitoring information regarding the distance between the user and the virtual object being close within a set distance to the artificial intelligence model (702).

[0218] The notification management unit (414) can determine whether the distance between a user wearing a wearable electronic device (101, 400) and a virtual object is located far apart, exceeding a certain distance. Since a notification is required when the distance between the user and the virtual object exceeds a certain distance (e.g., reference distance) and the user cannot see the virtual object (e.g., an angle at which the virtual object cannot be recognized), the notification management unit (414) can provide monitoring information to the artificial intelligence model (702) regarding the distance between the user and the virtual object exceeding a certain distance and the user cannot see the virtual object.

[0219] For example, the notification management unit (414) can determine whether a virtual object is positioned so that it overlaps with a user wearing a wearable electronic device (101, 400). Since a notification is required when the user and the virtual object are positioned so that they overlap, the notification management unit (414) can provide monitoring information regarding the overlapping positioning of the user and the virtual object to the artificial intelligence model (702).

[0220] For example, the notification management unit (414) can determine whether the direction of gaze of the user wearing the wearable electronic device (101, 400) is an angle at which the virtual object cannot be seen (e.g., an angle at which the virtual object cannot be recognized). Since a notification is required when the direction of gaze of the user wearing the wearable electronic device (101, 400) is an angle at which the virtual object cannot be seen (e.g., an angle at which the virtual object cannot be recognized), the notification management unit (414) can provide monitoring information regarding the direction of gaze of the user being at an angle at which the virtual object cannot be seen to the artificial intelligence model (702).

[0221] For example, the notification management unit (414) can determine that a virtual object viewed by a user wearing a wearable electronic device (101, 400) is obscured by a real object (e.g., person, animal, building, car) or another virtual object. Since a notification is required when a virtual object viewed by the user is obscured by a real object (e.g., person, animal, building, car) or another virtual object, the notification management unit (414) can provide monitoring information regarding the obscuring of a virtual object viewed by the user by a real object (e.g., person, animal, building, car) or another virtual object to an artificial intelligence model (702).

[0222] For example, the notification management unit (414) can provide monitoring information regarding the user looking at multiple virtual objects in a world lock state to the artificial intelligence model (702) when a user wearing the wearable electronic device (101, 400) looks at multiple virtual objects in a world lock state. The world lock state of the virtual objects of the wearable electronic device (101, 400) may include the virtual object displayed on the display (250) being in a locked state.

[0223] The world lock state of a virtual object of a wearable electronic device (101, 400) may include a state in which the virtual object is fixed at a certain position within a virtual world displayed on a display (250).

[0224] It indicates that the object displayed on the virtually displayed screen is in a locked state, and this is to be reflected in the explanation.

[0225] For example, the notification management unit (414) can determine that a user wearing a wearable electronic device (101, 400) is looking at a virtual messenger. Since a notification is required when the user is looking at the virtual messenger, the notification management unit (414) can provide monitoring information about the user looking at the virtual messenger to the artificial intelligence model (702).

[0226] For example, the notification management unit (414) can determine that a user wearing a wearable electronic device (101, 400) is watching a video with subtitles (e.g., movies, performances, news, documentaries, and other content with subtitles). Since a notification is required when the user is watching a video with subtitles (e.g., movies, performances, news, documentaries, and other content with subtitles), the notification management unit (414) can provide monitoring information regarding the user watching a video with subtitles (e.g., movies, performances, news, documentaries, and other content with subtitles) to an artificial intelligence model (702).

[0227] For example, the notification management unit (414) can determine that a user wearing the wearable electronic device (101, 400) is watching a video that is not included (e.g., movies, performances, news, documentaries, and other content with subtitles). Since a notification is required when the user is watching a video that is not included (e.g., movies, performances, news, documentaries, and other content with subtitles), the notification management unit (414) can provide monitoring information regarding the user watching a video that is not included (e.g., movies, performances, news, documentaries, and other content with subtitles) to the artificial intelligence model (702).

[0228] For example, the notification management unit (414) can determine that a user wearing a wearable electronic device (101, 400) is looking at a video containing financial information (e.g., an image, a document). Since a notification is required when the user is looking at a video containing financial information (e.g., an image, a document), the notification management unit (414) can provide monitoring information regarding the user looking at a video containing financial information (e.g., an image, a document) to the artificial intelligence model (702).

[0229] For example, the notification management unit (414) can determine that a user wearing a wearable electronic device (101, 400) is playing a game (or watching a game video). Since a notification is required when the user is playing a game (or watching a game video), the notification management unit (414) can provide (e.g., transmit) monitoring information about the user playing a game (or watching a game video) to an artificial intelligence model (702).

[0230] According to one embodiment, a wearable electronic device (101, 400) may include hardware and software specialized for processing an artificial intelligence model (702, AI model). For example, the artificial intelligence model (702, AI model) may be an LLM model or an LMM (LARGE MULIMODAL MODEL) model trained in various forms. Text, images, XML data, HTML, and electronic device notification information may be transmitted to the LLM model or LMM model. Information obtained using the LLM model or LMM model (e.g., information received) may be summarized, and the summarized information may be obtained. Information obtained using the LLM model or LMM model and the summarized information (e.g., text or images) may be provided to the user as audio output or display output (optional).

[0231] For example, an artificial intelligence model (702, AI model) may include neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above.

[0232] For example, the artificial intelligence model (702, AI model) can be embedded in a wearable electronic device (101, 400) or embedded in an external server.

[0233] According to one embodiment, the artificial intelligence model (702, AI model) can obtain at least one monitoring information from the notification management unit (414). For example, the artificial intelligence model (702, AI model) can obtain (e.g., receive) monitoring information from the notification management unit (414) regarding a user being located between a plurality of virtual objects.

[0234] For example, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding a user being positioned between multiple virtual objects, so as to change at least one of the shapes of one or more virtual objects, such as size and position, when the user is positioned between multiple virtual objects, and / or convert information of the multiple virtual objects into voice (e.g., sound) and output it.

[0235] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding the distance between the user and the virtual object being located within a certain distance, so as to change at least one of the shapes of the size, location, etc. of one or more virtual objects and / or convert information of the multiple virtual objects into voice (e.g., sound) and output it.

[0236] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information that the distance between the user and the virtual object exceeds a certain distance, so that the user cannot see the virtual object, and can change at least one of the shapes of the size, location, etc. of one or more virtual objects and / or convert information of the multiple virtual objects into voice (e.g., sound) and output it.

[0237] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding the positioning of a user and a virtual object overlapping, so as to change at least one of the shapes of one or more virtual objects, such as size and position, and / or convert information of the multiple virtual objects into voice (e.g., sound) and output it.

[0238] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information that the direction of the user's gaze is at an angle where the virtual object cannot be seen, so as to change at least one of the shapes of one or more virtual objects, such as size and position, and / or convert information of the multiple virtual objects into voice (e.g., sound) and output it.

[0239] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding whether a virtual object viewed by a user is obscured by a real object (e.g., person, animal, building, car) or another virtual object, so as to change at least one of the shapes of one or more virtual objects, such as size and location, and / or convert information of the multiple virtual objects into voice (e.g., sound) and output it.

[0240] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding a user looking at a plurality of virtual objects in a world lock state, so as to change at least one of the shapes of one or more virtual objects, such as size and position, and / or convert information of the plurality of virtual objects into voice (e.g., sound) and output it.

[0241] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding the user looking at the virtual messenger, so as to provide a summary of the content displayed on the messenger or convert the content displayed on the messenger into voice (e.g., sound) and output it.

[0242] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding a user viewing a video containing subtitles (e.g., a movie, performance, news, documentary, or other content containing subtitles), so as to provide a summary of the content displayed in the subtitles or to convert the content displayed in the subtitles into voice (e.g., sound) and output it.

[0243] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding a user viewing a video without subtitles (e.g., a movie, performance, news, documentary, or other content with subtitles), so as to summarize and display the content of the video or convert the content of the video into voice (e.g., sound) and output it.

[0244] According to one embodiment, the artificial intelligence model (702) can generate user notification information to summarize and display financial information or to display changes in financial information based on monitoring information regarding the user viewing a video (e.g., image, document) containing financial information.

[0245] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding the user viewing a video (e.g., image, document) containing financial information, so as to summarize the financial information or convert the changed content of the financial information into voice (e.g., sound) and output it.

[0246] According to one embodiment, the artificial intelligence model (702) can generate user notification information based on monitoring information regarding the user playing the game (or watching the game video), so as to display game play information or convert game play information into voice (e.g., sound) and output it.

[0247] According to one embodiment, user notification information generated by the artificial intelligence model (702) can be provided to the processor (410).

[0248] For example, the processor (410) can change at least one of the shapes, sizes, and positions of one or more virtual objects and provide data of the changed virtual objects to the display (250).

[0249] For example, the processor (4100) can convert information of one or more virtual objects into voice data (e.g., acoustic data) and provide the voice data (e.g., acoustic data) of the virtual objects to the voice output unit (425).

[0250] According to one embodiment, the display (250) can display a virtual object in which at least one of the shapes, such as size and position, has been changed based on data of the virtual object obtained from the processor (410).

[0251] According to one embodiment, the voice output unit (425) can output information of a virtual object as voice (e.g., sound) based on voice data (e.g., sound data) obtained from the processor (410).

[0252] FIG. 7b is a drawing (700) showing a method of operation of a wearable electronic device according to an embodiment of the present disclosure.

[0253] Referring to FIGS. 4, FIGS. 7a, and FIGS. 7b, in operation 710, a wearable electronic device (101, 400) according to one embodiment of the present disclosure (e.g., the wearable electronic device (101) of FIGS. 2a to 3b) can determine whether a user is wearing the wearable electronic device (101, 400) (e.g., an HMD). If, as a result of the determination of operation 710, the user is wearing the wearable electronic device (101, 400) (e.g., an HMD), the wearable electronic device (101, 400) can perform operation 720.

[0254] In operation 720, the wearable electronic device (101, 400) can determine whether a virtual object in a world-locked state exists on a screen displayed on the display (250) (e.g., whether a world-locked virtual object is displayed).

[0255] If, as a result of the determination of operation 720, a virtual object in a world-locked state exists (e.g., a world-locked virtual object is displayed), the wearable electronic device (101, 400) can perform operation 730.

[0256] In operation 730, the wearable electronic device (101, 400) can determine the state between the user and the object.

[0257] According to one embodiment, the wearable electronic device (101, 400) can determine whether the user wearing the wearable electronic device (101, 400) and the virtual object are separated by more than a reference distance. If the user and the virtual object are separated by more than a reference distance, the wearable electronic device (101, 400) can perform operation 740.

[0258] According to one embodiment, the wearable electronic device (101, 400) can determine whether a user wearing the wearable electronic device (101, 400) and a virtual object are located close within a reference distance (e.g., a certain distance). If the user and the virtual object are located close within a reference distance (e.g., a certain distance), the wearable electronic device (101, 400) can perform operation 740.

[0259] According to one embodiment, the wearable electronic device (101, 400) can determine whether the direction of gaze of a user wearing the wearable electronic device (101, 400) is at an angle where the virtual object cannot be seen. If the direction of gaze of the user is at an angle where the virtual object cannot be seen, the wearable electronic device (101, 400) can perform operation 740.

[0260] In operation 740, the wearable electronic device (101, 400) can analyze information contained in a virtual object. For example, the wearable electronic device (101, 400) can analyze information contained in a virtual object using a pre-trained artificial intelligence model (702) and determine the user's context. The wearable electronic device (101, 400) can determine the size, location, and shape of the virtual object. The wearable electronic device (101, 400) can determine whether the virtual object is a person, animal, document, video, image, or messenger.

[0261] For example, when a wearable electronic device (101, 400) analyzes information contained in a virtual object, it can input the entire information or some content (updated content) of the screen into an artificial intelligence model (702) to check the summary result or whether there is information updated after the positional relationship between the user and the virtual object has changed.

[0262] For example, when a wearable electronic device (101, 400) analyzes information contained in a virtual object, it can input notification-related information of the app into an artificial intelligence model (702) to check whether there is a summary result or information updated after the position relationship between the user and the virtual object has changed.

[0263] In operation 750, the wearable electronic device (101, 400) can generate user notification information according to the characteristics of each virtual object and provide it to the user.

[0264] For example, if the results analyzed in Action 750 include content that can be notified to the user, the analyzed results can be provided to the user as an audio output.

[0265] For example, a wearable electronic device (101, 400) can generate user notification information by changing at least one of the shapes of each virtual object, such as size and position, and provide the generated user notification information to the user.

[0266] For example, a wearable electronic device (101, 400) can summarize information of each virtual object to generate user notification information and provide the generated user notification information to the user.

[0267] For example, a wearable electronic device (101, 400) can generate user notification information by summarizing the content displayed in the messenger and provide the generated user notification information to the user.

[0268] In operation 760, the wearable electronic device (101, 400) can convert information of each virtual object into voice (e.g., sound) data and output information of each virtual object as voice (e.g., sound) using a voice output unit (425).

[0269] For example, based on the analyzed results regarding content that can be notified to the user, screen information can be provided to the user as audio output.

[0270] For example, based on the analyzed results regarding content that can be notified to the user, screen information can be summarized and provided to the user as audio output.

[0271] For example, based on the analyzed results regarding content that can be notified to the user, an existing object can be modified and displayed, or a new object can be created and associated with that object to display the content.

[0272] For example, the wearable electronic device (101, 400) can generate user notification information to convert the content displayed in the messenger into voice (e.g., sound) and output the content of the messenger as voice (e.g., sound) using the voice output unit (425).

[0273] Instructions for performing operations 710 to 760 illustrated in FIG. 7b may be stored in memory (415). At least one processor (410), a notification management unit (414), and an AI model (720) may load the instructions stored in memory (415) to perform operations 710 to 760.

[0274] For example, the wearable electronic device (101, 400) can sequentially perform operations 710 to 760 as illustrated in FIG. 7b.

[0275] For example, a wearable electronic device (101, 400) can perform at least some of the operations 710 to 760 shown in FIG. 7b simultaneously (e.g., in parallel).

[0276] FIG. 8 relates to a method of operation of a wearable electronic device according to an embodiment of the present disclosure, and is a drawing showing the recognition of the distance and direction between a user and a virtual object and the display of a virtual object by changing the virtual object.

[0277] Referring to FIGS. 4, FIGS. 7a, and FIGS. 8, a wearable electronic device (101, 400) can recognize the distance between a user (830) and one or more virtual objects (840). The wearable electronic device (101, 400) can recognize the direction between the user (830) and one or more virtual objects (840).

[0278] For example, a wearable electronic device (101, 400) can determine whether one or more virtual objects (840) are smaller than or equal to a preset reference size. The wearable electronic device (101, 400) can determine whether the virtual object (840) is smaller than the reference size by comparing the horizontal size or vertical size, or area or width of the virtual object (840) with the preset reference size.

[0279] For example, a wearable electronic device (101, 400) can determine whether one or more virtual objects (840) are larger than a preset reference size. The wearable electronic device (101, 400) can determine whether the virtual object (840) is larger than the reference size by comparing the horizontal size or vertical size, or area or width of the virtual object (840) with the preset reference size.

[0280] The wearable electronic device (101, 400) can determine whether a user (830) is located between multiple virtual objects (840) of small size. If the user (830) is located between multiple virtual objects (840) of small size, the wearable electronic device (101, 400) can move and display the positions of the multiple virtual objects (850) so that the user does not overlap with the multiple virtual objects (840).

[0281] For example, the wearable electronic device (101, 400) can move the positions of multiple virtual objects (850) so that the user (830) and the multiple virtual objects (840) do not overlap when the user (830) moves. For example, the wearable electronic device (101, 400) can determine whether the distance between the user (830) and the small virtual objects (840) is within a predetermined distance when the user (830) moves. If the distance between the user (830) and the virtual objects (840) is within a predetermined distance, the electronic device (101, 400) can move the positions of the virtual objects (850) to avoid the user (830). For example, when the user (830) moves, if the user (830) is located between small virtual objects (840), the wearable electronic device (101, 400) can move the position of the virtual objects (850) to avoid the user (830).

[0282] FIG. 9 relates to a method of operation of a wearable electronic device according to an embodiment of the present disclosure, and is a drawing showing the recognition of the distance and direction between a user and a virtual object and the display of a virtual object by changing the virtual object.

[0283] Referring to FIG. 4, FIG. 7a, and FIG. 9, a wearable electronic device (101, 400) can determine whether a virtual object (940) is larger than a preset reference size. The wearable electronic device (101, 400) can determine whether the virtual object (940) is larger than the reference size by comparing the horizontal size or vertical size, or area or width of the virtual object (940) with the preset reference size.

[0284] For example, a wearable electronic device (101, 400) can determine whether a virtual object (940) is larger than a preset reference size. The wearable electronic device (101, 400) can determine whether the virtual object (940) is larger than the reference size by comparing the horizontal size or vertical size, or area or width of the virtual object (940) with the preset reference size.

[0285] For example, the wearable electronic device (101, 400) can recognize whether a large virtual object (940) is located in the direction in which the user (830) is moving. The wearable electronic device (101, 400) can recognize the distance between the user (930) and the large virtual object (940).

[0286] The wearable electronic device (101, 400) can display the large virtual object (950) by separating it into multiple parts when the distance between the user (930) and the large virtual object (940) is close to a reference distance or overlaps with each other. For example, the wearable electronic device (101, 400) can separate the large virtual object (950) into two parts before the user (930) and the large virtual object (940) overlap, so that the user (930) and the virtual object (850) separated into two parts do not overlap.

[0287] FIG. 10 illustrates the operation of a wearable electronic device according to an embodiment of the present disclosure, and is a drawing showing how to change and display a virtual object when a virtual object and a real object overlap.

[0288] Referring to FIG. 4, FIG. 7a, and FIG. 10, a wearable electronic device (101, 400) can display a real object (1040) (e.g., a car) and one or more virtual objects (1050) on a display (250).

[0289] For example, a wearable electronic device (101, 400) can determine whether one or more virtual objects (1040) are smaller than a preset reference size. The wearable electronic device (101, 400) can determine whether the virtual object (1040) is smaller than the reference size by comparing the horizontal size or vertical size, or area or width of the virtual object (1040) with the preset reference size.

[0290] For example, a wearable electronic device (101, 400) can determine whether one or more virtual objects (1040) are larger than a preset reference size. The wearable electronic device (101, 400) can determine whether the virtual object (1040) is larger than the reference size by comparing the horizontal size or vertical size, or area or width of the virtual object (1040) with the preset reference size.

[0291] According to one embodiment, a wearable electronic device (101, 400) can display one or more virtual objects (1050) having a size smaller than a real object (1040) (e.g., a car) on a display (250).

[0292] According to one embodiment, in operation 1010, the wearable electronic device (101, 400) can recognize the gaze of the user (1030) and obtain the range of the gaze of the user (1030). The wearable electronic device (101, 400) can determine whether, within the range of the user's (1030) gaze, one or more virtual objects (1050) that are smaller in size than the actual object (1040) (e.g., a car) are obscuring the actual object (1040) (e.g., a car).

[0293] According to one embodiment, in operation 1020, when one or more virtual objects (1050) that are smaller in size than a real object (1040) (e.g., a car) in the line of sight of a user (1030) obscure the real object (1040) (e.g., a car), the wearable electronic device (101, 400) can display the position of one or more virtual objects (1060) by moving them so that the one or more virtual objects (1050) do not overlap with the real object (1040) (e.g., a car).

[0294] For example, when the position of one or more virtual objects (1060) is changed and obscured by a real object (1040) (e.g., a car), the wearable electronic device (101, 400) can convert information of the one or more virtual objects (1060) obscured by the real object (1040) (e.g., a car) into voice (e.g., sound) and output it.

[0295] According to one embodiment, in operation 1020, when one or more virtual objects (1050) that are smaller in size than a real object (1040) (e.g., a car) in the line of sight of a user (1030) obscure the real object (1040) (e.g., a car), the wearable electronic device (101, 400) may display one or more virtual objects (1060) separately so that the one or more virtual objects (1050) do not overlap with the real object (1040) (e.g., a car).

[0296] For example, when a virtual object (1060) is displayed separately, information of the separated virtual object (1060) can be converted into voice (e.g., sound) and output. FIG. 11 is a diagram showing the operation of a wearable electronic device according to an embodiment of the present disclosure, which recognizes the distance and direction between a user and a virtual object and changes and displays the virtual object.

[0297] Referring to FIG. 4, FIG. 7a, and FIG. 11, a wearable electronic device (101, 400) can display a real object (1140) (e.g., a car) and one or more virtual objects (1150) on a display (250).

[0298] For example, a wearable electronic device (101, 400) can determine whether one or more virtual objects (1140) are smaller than or equal to a preset reference size. The wearable electronic device (101, 400) can determine whether the virtual object (1140) is smaller than the reference size by comparing the horizontal size or vertical size, or area or width of the virtual object (1140) with the preset reference size.

[0299] For example, a wearable electronic device (101, 400) can determine whether one or more virtual objects (1140) are larger than a preset reference size. The wearable electronic device (101, 400) can determine whether the virtual object (1140) is larger than the reference size by comparing the horizontal size or vertical size, or area or width of the virtual object (1140) with the preset reference size.

[0300] A wearable electronic device (101, 400) can display one or more virtual objects (1150) on a display (250) that are similar in size to a real object (1140) (e.g., a car) or larger in size than a real object (1140) (e.g., a car).

[0301] According to one embodiment, in operation 1110, the wearable electronic device (101, 400) can recognize the gaze of the user (1130) and obtain the gaze range of the user (1130). Within the gaze range of the user (1130), the wearable electronic device (101, 400) can recognize the location of a real object (1140) (e.g., a car) and a virtual object (1150) in the direction in which the user (830) is moving. The wearable electronic device (101, 400) can recognize the distance between the real object (1140) (e.g., a car) and the virtual object (1150).

[0302] According to one embodiment, in operation 1120, when the distance between a real object (1140) (e.g., a car) and a virtual object (1150) is close to or less than a reference distance or overlaps with each other, the wearable electronic device (101, 400) can separate the virtual object (1160) into a plurality of parts and display them on a display (250).

[0303] For example, a wearable electronic device (101, 400) can separate a virtual object (1160) into two parts before a real object (1140) (e.g., a car) and a virtual object (1150) overlap each other, so that the real object (1140) (e.g., a car) and the virtual object (1160) separated into two parts do not overlap.

[0304] For example, in the case where there is a part of a virtual object (1160) separated into two parts that is obscured by a real object (1140) (e.g., a car), the wearable electronic device (101, 400) can convert information about the part of the virtual object (1160) that is obscured by the real object (1140) (e.g., a car) into voice (e.g., sound) and output it.

[0305] FIG. 12 is a drawing (1200) showing the operation of a wearable electronic device according to an embodiment of the present disclosure.

[0306] Referring to FIGS. 4, FIGS. 7a, and FIGS. 12, in operation 1210, a wearable electronic device (101, 400) according to one embodiment of the present disclosure (e.g., the wearable electronic device (101) of FIGS. 2a to 3b) can determine whether a virtual object in a world-locked state exists on the screen of a display (250). For example, the wearable electronic device (101, 400) can determine whether a virtual object in a world-locked state is displayed on the display (250). If, as a result of the determination of operation 1210, a virtual object in a world-locked state exists on the screen of the display (250), the wearable electronic device (101, 400) can perform operation 1220.

[0307] In operation 1220, the wearable electronic device (101, 400) can determine whether the distance between the user and the virtual object is closer than a reference distance, or whether the user and the virtual object are overlapping. The wearable electronic device (101, 400) can determine whether the distance between another object (e.g., a real object, another virtual object) and the virtual object is closer than a reference distance, or whether the virtual object is overlapping with another object (e.g., a real object, another virtual object).

[0308] If, as a result of the judgment of operation 1220, the distance between the user and the virtual object is closer than the reference distance, or if the user and the virtual object are in an overlapping state, the wearable electronic device (101, 400) can perform operation 1230.

[0309] If, as a result of the judgment of operation 1220, the distance between another object (e.g., a real object, another virtual object) and the virtual object is closer than the reference distance, or if the virtual object and another object (e.g., a real object, another virtual object) overlap, the wearable electronic device (101, 400) can perform operation 1230.

[0310] In operation 1230, the wearable electronic device (101, 400) can move the position of the virtual object so that the user and the virtual object do not overlap. The wearable electronic device (101, 400) can move the position of the virtual object so that the virtual object and other objects (e.g., real objects, other virtual objects) do not overlap.

[0311] In operation 1240, the wearable electronic device (101, 400) can determine whether the user stays for a certain period of time in the area (e.g., location) where operation 1220 was performed.

[0312] If, as a result of the judgment of operation 1240, the user moves (e.g., moves away) from the area (e.g., location) where operation 1220 was being performed, the wearable electronic device (101, 400) can perform operation 1250.

[0313] In operation 1250, the wearable electronic device (101, 400) can move the virtual object that was moved to a different location in operation 1230 back to its original location (e.g., restored) when the distance between the user and the virtual object becomes greater than a reference distance.

[0314] If, as a result of the judgment of operation 1240, the user stays for a certain period of time in the area (e.g., location) where operation 1220 was performed, the wearable electronic device (101, 400) can perform operation 1260.

[0315] In operation 1260, the wearable electronic device (101, 400) can analyze information contained in a virtual object. For example, the wearable electronic device (101, 400) can analyze information contained in a virtual object using a pre-trained artificial intelligence model (e.g., the artificial intelligence model (702) of FIG. 7) and can determine the user's context. The wearable electronic device (101, 400) can determine the size, location, and shape of the virtual object. The wearable electronic device (101, 400) can determine whether the virtual object is a person, animal, document, video, image, or messenger.

[0316] For example, when a wearable electronic device (101, 400) analyzes information contained in a virtual object, it can input the entire information or some content (updated content) of the screen into an artificial intelligence model (702) to check whether there is a summary result or information updated after the positional relationship between the user and the virtual object has changed.

[0317] For example, when a wearable electronic device (101, 400) analyzes information contained in a virtual object, it can input notification-related information of the app into an artificial intelligence model (702) to check whether there is a summary result or information updated after the position relationship between the user and the virtual object has changed.

[0318] For example, the wearable electronic device (101, 400) can identify the characteristics of each virtual object and the user's context using a pre-trained artificial intelligence model (702). The wearable electronic device (101, 400) can identify the size, location, and shape of the virtual object. The wearable electronic device (101, 400) can identify whether the virtual object is a person, animal, document, video, image, or messenger. After performing operation 1260, the wearable electronic device (101, 400) can perform operations 750 to 760 illustrated in FIG. 7b.

[0319] Instructions for performing operations 1210 to 1260 illustrated in FIG. 12 may be stored in memory (415). At least one processor (410), a notification management unit (414), and an AI model (720) may load the instructions stored in memory (415) to perform operations 1210 to 1260.

[0320] For example, a wearable electronic device (101, 400) can sequentially perform operations 1210 to 1260 as illustrated in FIG. 12.

[0321] For example, a wearable electronic device (101, 400) can perform at least some of the operations 1210 to 1260 shown in FIG. 12 simultaneously (e.g., in parallel).

[0322] FIG. 13 illustrates the operation of a wearable electronic device according to an embodiment of the present disclosure, showing that information of a virtual object is output as voice when the direction of the user's gaze is at an angle where the virtual object cannot be seen.

[0323] Referring to FIG. 4, FIG. 7a and FIG. 13, a wearable electronic device (101, 400) can display a virtual object (1340) on a display (250).

[0324] In operation 1310, the wearable electronic device (101, 400) can determine whether the user (1330) is looking at a virtual object. The wearable electronic device (101, 400) can track the user's (1330) gaze and determine an area perceptible to the user's (1330) gaze.

[0325] For example, the wearable electronic device (101, 400) can recognize the gaze of the user (1330) and obtain the range of gaze that the user (1330) is looking at. The wearable electronic device (101, 400) can determine whether, within the range of gaze of the user (1330), the direction of the gaze that the user (1330) is looking at corresponds to an angle in which the virtual object (1340) cannot be seen (e.g., an angle in which the virtual object cannot be recognized).

[0326] For example, the wearable electronic device (101, 400) can determine whether the user (1330) can perceive the virtual object or the contents of the virtual object based on whether the user's (1330) gaze is directed toward the virtual object.

[0327] For example, the wearable electronic device (101, 400) can determine whether the user (1330) can perceive the virtual object or the contents of the virtual object based on whether the virtual object exists within a pre-specified area based on the gaze. For example, if the location of the virtual object (1340) is outside the angle of the gaze of the user (1330), the wearable electronic device (101, 400) can determine that the user (1330) cannot see the virtual object (1340). If the direction of the gaze of the user (1330) corresponds to an angle where the virtual object (1340) cannot be seen (e.g., if the angle between the point where the gaze is directed and the location of the virtual object is outside the set range), the wearable electronic device (101, 400) can perform operation 1320.

[0328] In operation 1320, when the user (1330) moves their gaze in a direction where the virtual object (1340) cannot be recognized, or when the distance from the virtual object (1340) increases so that the virtual object (1340) cannot be recognized, the wearable electronic device (101, 400) can convert information of the virtual object (1340) into voice (e.g., sound) and output it.

[0329] For example, if the direction of the user's (1330) gaze corresponds to an angle at which the virtual object (1340) cannot be seen (e.g., an angle at which the virtual object cannot be recognized), the wearable electronic device (101, 400) can convert the information of the virtual object (1340) into voice (e.g., sound) and output it.

[0330] For example, if the direction of the user's (1330) gaze corresponds to an angle at which the virtual object (1340) cannot be seen (e.g., an angle at which the virtual object cannot be recognized), the wearable electronic device (101, 400) can convert information about the virtual object (1340) changing into voice (e.g., sound) and output it.

[0331] FIG. 14 illustrates the operation of a wearable electronic device according to an embodiment of the present disclosure, and is a diagram showing that information of a virtual object is output as voice when a user and a virtual object overlap.

[0332] Referring to FIG. 4, FIG. 7a and FIG. 14, a wearable electronic device (101, 400) can display a virtual object (1440) on a display (250).

[0333] The wearable electronic device (101, 400) can recognize the gaze of the user (1430) and obtain the range of the gaze of the user (1430). The wearable electronic device (101, 400) can determine whether the distance between the user (1430) and the virtual object (1440) is within a reference distance within the range of the user's (1430) gaze, or whether the user (1430) and the virtual object (1440) overlap.

[0334] For example, when a user (1430) and a virtual object (1440) overlap, the wearable electronic device (101, 400) can perform operation 1410.

[0335] For example, when the distance between the user (1430) and the virtual object (1440) is within a reference distance, the wearable electronic device (101, 400) can perform operation 1420.

[0336] In operation 1410, when the user (1430) and the virtual object (1440) overlap, the wearable electronic device (101, 400) can convert the information of the virtual object (1440) into voice (e.g., sound) and output it. For example, when the user (1430) and the virtual object (1440) overlap, the wearable electronic device (101, 400) can summarize the information of the virtual object (1440) and convert the summarized information of the virtual object (1440) into voice (e.g., sound) and output it.

[0337] In operation 1420, when the distance between the user (1430) and the virtual object (1440) is within a reference distance, the wearable electronic device (101, 400) can convert the information of the virtual object (1440) into voice (e.g., sound) and output it.

[0338] For example, when the distance between the user (1430) and the virtual object (1440) is within a reference distance, the wearable electronic device (101, 400) can summarize the information of the virtual object (1440) and convert the summarized information of the virtual object (1440) into voice (e.g., sound) and output it.

[0339] FIG. 15 illustrates the operation of a wearable electronic device according to an embodiment of the present disclosure, and is a diagram showing that information of a virtual object is output as voice when the user cannot look at the virtual object.

[0340] Referring to FIG. 4, FIG. 7a and FIG. 15, in operation 1510, the wearable electronic device (101, 400) can display a virtual object (1540) on a display (250).

[0341] For example, the wearable electronic device (101, 400) can recognize the gaze of the user (1530) and obtain the range of the gaze of the user (1530). The wearable electronic device (101, 400) can determine whether the direction of the user's (1530) gaze corresponds to an angle in which the virtual object (1540) cannot be seen (e.g., an angle in which the virtual object cannot be recognized).

[0342] For example, if the location of the virtual object (1440) is outside the angle of the user's (1530) gaze, the wearable electronic device (101, 400) may determine that the user (1530) cannot see the virtual object (1540). If the direction of the user's (1530) gaze is outside the angle at which the virtual object (1540) cannot be seen (e.g., the angle between the point where the gaze is directed and the location of the virtual object is outside the set range), the wearable electronic device (101, 400) may convert the information of the virtual object (1540) into voice (e.g., sound) and output it (1550).

[0343] According to one embodiment, a wearable electronic device (101, 400) can perceive the distance between a user (1530) and a virtual object (1540).

[0344] For example, if the distance between the user (1530) and the virtual object (1540) is greater than a reference distance and the user cannot perceive the virtual object (1540), the wearable electronic device (101, 400) can convert the information of the virtual object (1540) into voice (e.g., sound) and output it (1550).

[0345] According to one embodiment, in operation 1520, the wearable electronic device (101, 400) can display a virtual object (1560) on a display (250).

[0346] For example, the wearable electronic device (101, 400) can recognize the gaze of the user (1530) and obtain the range of the gaze of the user (1530). When the user (1530)’s gaze direction is toward the edge of the virtual object (1560), the wearable electronic device (101, 400) may not be able to accurately see the information of the edge of the virtual object (1560).

[0347] For example, if the user (1530) cannot accurately see the information of the edges of the virtual object (1560) due to the direction of the user's gaze, the wearable electronic device (101, 400) can convert the information of the virtual object (1560) into voice (e.g., sound) and output it.

[0348] For example, if the user (1530)'s gaze direction cannot accurately see the information of the edges of the virtual object (1560), the wearable electronic device (101, 400) can summarize the information of the virtual object (1560). The wearable electronic device (101, 400) can convert the summarized information of the virtual object (1560) into voice (e.g., sound) and output it.

[0349] FIG. 16 is a drawing (1600) showing the operation of a wearable electronic device according to an embodiment of the present disclosure.

[0350] Referring to FIG. 4, FIG. 7a and FIG. 16, in operation 1610, the wearable electronic device (101, 400) can determine whether the distance between the user and the virtual object is closer than a reference distance, or whether the user and the virtual object are in an overlapping state.

[0351] If, as a result of the determination of operation 1610, the distance between the user and the virtual object is closer than the reference distance, or if the user and the virtual object are in an overlapping state, the wearable electronic device (101, 400) can perform operations 1620 to 1660.

[0352] In operation 1620, the wearable electronic device (101, 400) can determine whether there is one virtual object displayed on the display (250).

[0353] In operation 1630, when there is only one virtual object displayed on the display (250), the wearable electronic device (101, 400) can perform operations 730 to 760 of FIG. 7 to convert the information of the virtual object into voice (e.g., sound) and output it.

[0354] For example, a wearable electronic device (101, 400) can summarize information about a virtual object (1560). The wearable electronic device (101, 400) can convert the summary information of the virtual object (1560) into voice (e.g., sound) and output it.

[0355] In operation 1640, it can be determined whether there are multiple virtual objects displayed on the display (250).

[0356] In operation 1650, when there are multiple virtual objects displayed on the display (250), the wearable electronic device (101, 400) can determine the priority of the multiple virtual objects.

[0357] For example, a wearable electronic device (101, 400) can determine the priority of multiple virtual objects within a preset maximum number.

[0358] For example, if there is a virtual object displaying financial information (e.g., stock price information) among multiple virtual objects, the virtual object displaying financial information (e.g., stock price information) can be determined as the first priority (e.g., highest priority).

[0359] For example, if there is a virtual object displaying real-time news among multiple virtual objects, the virtual object displaying real-time news can be determined as the second priority.

[0360] In operation 1660, the wearable electronic device (101, 400) can perform operations 730 to 760 of FIG. 7 and convert information of each virtual object into voice (e.g., sound) and output it according to a determined priority.

[0361] For example, the wearable electronic device (101, 400) can summarize information of each virtual object according to a determined priority. The wearable electronic device (101, 400) can convert the summary information of each virtual object (1560) into voice (e.g., sound) and output it.

[0362] For example, the wearable electronic device (101, 400) can convert information about the current stock price contained in a virtual object into voice (e.g., sound) and output it.

[0363] For example, the wearable electronic device (101, 400) can convert information about fluctuations in stock prices contained in a virtual object into voice (e.g., sound) and output it.

[0364] For example, a wearable electronic device (101, 400) can convert information about real-time news contained in a virtual object into voice (e.g., sound) and output it.

[0365] Instructions for performing operations 1610 to 1660 illustrated in FIG. 16 may be stored in memory (415). At least one processor (410), a notification management unit (414), and an AI model (720) may load the instructions stored in memory (415) to perform operations 1610 to 1660.

[0366] For example, a wearable electronic device (101, 400) can sequentially perform operations 1610 to 1660 as illustrated in FIG. 16.

[0367] For example, a wearable electronic device (101, 400) can perform at least some of the operations 1610 to 1660 shown in FIG. 16 simultaneously (e.g., in parallel).

[0368] FIG. 17 is a drawing showing the operation of a wearable electronic device according to an embodiment of the present disclosure.

[0369] Referring to FIG. 4, FIG. 7a and FIG. 17, in operation 171, the wearable electronic device (101, 400) can perform operations 710 to 730 as illustrated in FIG. 7.

[0370] In operation 1720, the wearable electronic device (101, 400) can analyze information contained in a virtual object. For example, the wearable electronic device (101, 400) can analyze information contained in a virtual object using a pre-trained artificial intelligence model (e.g., the artificial intelligence model (702) of FIG. 7) and can determine the user's context. The wearable electronic device (101, 400) can determine the size, location, and shape of the virtual object. The wearable electronic device (101, 400) can determine whether the virtual object is a person, animal, document, video, image, or messenger.

[0371] For example, when a wearable electronic device (101, 400) analyzes information contained in a virtual object, it can input the entire information or some content (updated content) of the screen into an artificial intelligence model (702) to check the summary result or whether there is information updated after the positional relationship between the user and the virtual object has changed.

[0372] For example, when a wearable electronic device (101, 400) analyzes information contained in a virtual object, it can input notification-related information of the app into an artificial intelligence model (702) to check whether there is a summary result or information updated after the position relationship between the user and the virtual object has changed.

[0373] For example, the wearable electronic device (101, 400) can analyze information contained in a virtual object. For example, the wearable electronic device (101, 400) can identify the characteristics of each virtual object and the user's context using a pre-trained artificial intelligence model (702). The wearable electronic device (101, 400) can identify the size, location, and shape of the virtual object. The wearable electronic device (101, 400) can identify whether the virtual object is a person, animal, document, video, image, or messenger.

[0374] In operation 1730, the wearable electronic device (101, 400) can summarize the information of the conversation displayed in the messenger by understanding the conversation context of the messenger chat room with newly generated updates based on the characteristics of the virtual object (1720) when the virtual object is recognized as a messenger.

[0375] In operation 1740, the wearable electronic device (101, 400) can convert information of a conversation displayed in a messenger into voice (e.g., sound).

[0376] For example, a wearable electronic device (101, 400) can summarize information of a conversation displayed in a messenger and convert the summary information of the conversation into voice (e.g., sound).

[0377] In operation 1750, the wearable electronic device (101, 400) can operate the voice output unit (425) to output information of a conversation displayed on the messenger as voice (e.g., sound). For example, the wearable electronic device (101, 400) can operate the voice output unit (425) to output summary information of a conversation displayed on the messenger as voice (e.g., sound).

[0378] Instructions for performing operations 1710 to 1750 illustrated in FIG. 17 may be stored in memory (415). At least one processor (410), a notification management unit (414), and an AI model (720) may load the instructions stored in memory (415) to perform operations 1710 to 1750.

[0379] For example, a wearable electronic device (101, 400) can sequentially perform operations 1710 to 1750 as illustrated in FIG. 17.

[0380] For example, a wearable electronic device (101, 400) can perform at least some of the operations 1710 to 1750 shown in FIG. 17 simultaneously (e.g., in parallel).

[0381] FIG. 18 is a drawing showing the operation of a wearable electronic device according to an embodiment of the present disclosure.

[0382] In operation 1805, the wearable electronic device (101, 400) can perform operations 710 to 730 of FIG. 7.

[0383] In operation 1810, the wearable electronic device (101, 400) can determine whether the virtual object is a pre-specified application. For example, the wearable electronic device (101, 400) can determine whether the virtual object is an application that performs video content playback.

[0384] For example, the wearable electronic device (101, 400) can analyze information contained in a virtual object. For example, the wearable electronic device (101, 400) can analyze information contained in a virtual object using a pre-trained artificial intelligence model (e.g., the artificial intelligence model (702) of FIG. 7) and can identify the user's context. The wearable electronic device (101, 400) can identify the size, location, and shape of the virtual object. The wearable electronic device (101, 400) can identify whether the virtual object is a person, animal, document, video, image, messenger, or an application that plays video content.

[0385] For example, when a wearable electronic device (101, 400) analyzes information contained in a virtual object, it can input the entire information or some content (updated content) of the screen into an artificial intelligence model (702) to check whether there is a summary result or information updated after the positional relationship between the user and the virtual object has changed.

[0386] For example, when a wearable electronic device (101, 400) analyzes information contained in a virtual object, it can input notification-related information of the app into an artificial intelligence model (702) to check whether there is a summary result or information updated after the position relationship between the user and the virtual object has changed.

[0387] For example, the wearable electronic device (101, 400) can identify the characteristics of each virtual object and the user's context using a pre-trained artificial intelligence model (702). The wearable electronic device (101, 400) can identify the size, location, and shape of the virtual object. The wearable electronic device (101, 400) can identify whether the virtual object is a person, animal, document, video, image, messenger, or an application that plays video content.

[0388] In operation 1815, the wearable electronic device (101, 400) can recognize a virtual object as video content being played.

[0389] In operation 1820, the wearable electronic device (101, 400) can determine whether a virtual object (e.g., video content being played) is being played along with voice information.

[0390] If, as a result of the judgment of operation 1820, a virtual object (e.g., video content being played) is not played along with audio information, operation 1825 can be performed.

[0391] In operation 1825, the wearable electronic device (101, 400) can summarize the video change situation of the video content being played as voice (e.g., sound) information.

[0392] In operation 1830, the wearable electronic device (101, 400) can operate the voice output unit (425) to output information about the changing situation of the video content being played as voice (e.g., sound).

[0393] If, as a result of the judgment of operation 182, a virtual object (e.g., video content being played) is played along with audio information, operation 1835 can be performed.

[0394] In operation 1835, the wearable electronic device (101, 400) can determine whether the voice information of the video content being played is in a language set by the first user (e.g., User A). If the voice information of the video content being played is in a language set by the first user (e.g., User A), operation 1840 can be performed.

[0395] In operation 1840, the wearable electronic device (101, 400) can determine whether subtitles in a different language are being played together with the first voice information (e.g., voice information A) of the video content being played.

[0396] If, as a result of the judgment of operation 1840, the first audio information (e.g., audio information A) of the video content being played and subtitles in a different language are played together, operation 1845 can be performed.

[0397] In operation 1845, the wearable electronic device (101, 400) can analyze the first voice information (e.g., voice information A) of the video content being played and generate the second voice information (e.g., voice information B) with the voices of the characters added to the subtitles.

[0398] In operation 1850, the wearable electronic device (101, 400) converts the voice rare of the first voice information (e.g., voice information A) into voice information A2 with the voice rare silenced.

[0399] In operation 1855, the wearable electronic device (101, 400) can insert a second voice information (e.g., voice information B) into a silent portion in accordance with the playback speed of voice information A2, and operate the voice output unit (425) to output the second voice information (e.g., voice information B) as voice (e.g., sound).

[0400] Instructions for performing operations 1805 to 1855 illustrated in FIG. 18 may be stored in memory (415). At least one processor (410), a notification management unit (414), and an AI model (720) may load the instructions stored in memory (415) to perform operations 1805 to 1855.

[0401] For example, a wearable electronic device (101, 400) can sequentially perform operations 1805 to 1855 shown in FIG. 18.

[0402] For example, a wearable electronic device (101, 400) can perform at least some of the operations 1805 to 1855 shown in FIG. 18 simultaneously (e.g., in parallel).

[0403] The wearable electronic device and the method of operation thereof according to the embodiment of the present disclosure can improve the safety of use and user experience of the wearable electronic device by changing and displaying the size, position, and shape of a virtual object.

[0404] The wearable electronic device and the method of operation thereof according to the embodiment of the present disclosure can enable a user to perceive information about a virtual object without missing it.

[0405] The wearable electronic device and the method of operation according to the embodiment of the present disclosure can reduce visual clutter in a virtual space and increase space utilization by moving the position of a virtual object according to the distance between the user's gaze direction and the virtual object.

[0406] A wearable electronic device and a method of operating the same according to an embodiment of the present disclosure can reduce resources required for information transmission by summarizing meaningful information among information of a plurality of virtual objects and providing summary information of virtual objects to a user.

[0407] A wearable electronic device and a method of operation thereof according to an embodiment of the present disclosure can output information of a virtual object as voice (e.g., sound) when the distance between the user and the virtual object is too close or too far apart.

[0408] The wearable electronic device and the method of operation thereof according to the embodiment of the present disclosure can change and display the position of a virtual object when a user and a virtual object overlap.

[0409] A wearable electronic device and a method of operation thereof according to an embodiment of the present disclosure can output information of a virtual object as voice (e.g., sound) when a user and a virtual object overlap.

[0410] A wearable electronic device and a method of operation thereof according to an embodiment of the present disclosure can output information of an object as voice (e.g., sound) when the user's gaze direction is different from the location where a virtual object is displayed.

[0411] The wearable electronic device and the method of operation thereof according to the embodiment of the present disclosure can change and display the position of a virtual object when a real object and a virtual object overlap.

[0412] A wearable electronic device and a method of operation thereof according to an embodiment of the present disclosure can output information of a virtual object as voice (e.g., sound) when a real object and a virtual object overlap.

[0413] The wearable electronic device and the method of operation thereof according to an embodiment of the present disclosure can display a virtual object by dividing it into a plurality of parts so that the user and the virtual object do not overlap when a real object and a virtual object overlap.

[0414] In addition, various effects identified directly or indirectly through this document may be provided.

[0415] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

Claims

1. In a wearable electronic device (101, 400), A display (250) that displays external real objects and virtual objects on a single screen; A display driving unit that controls the driving of the above display (250); An audio module that converts electrical signals into sound and outputs them (voice output unit (425)); At least one processor (410) that controls the operation of the display driving unit and the audio module (voice output unit (425)); and It includes a memory (415) for storing instructions, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, A virtual object is displayed at a first position spaced a first distance in a first direction from a reference point, and The local location of the wearable electronic device (101, 400) is set as a reference point, and If the user's gaze from the current location is in a direction where the virtual object cannot be seen, Providing information about the above virtual object to the user via voice, Wearable electronic device (101, 400).

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, When multiple virtual objects exist, the priority of the multiple virtual objects is set, and Providing information about the plurality of virtual objects to the user in voice based on the set priority, Wearable electronic device.

3. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, If the user and the virtual object are separated by more than a certain distance from the current location, or are close within a certain distance, information about the virtual object is provided to the user by voice. Wearable electronic device (101, 400).

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, When the above virtual object overlaps with a real object, information about the above virtual object is provided to the user via voice. Wearable electronic device (101, 400).

5. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, When the virtual object overlaps with a real object, the virtual object is moved and displayed so that the virtual object does not overlap with the real object. Wearable electronic device.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, When the virtual object overlaps with a real object, the virtual object is separated into at least two parts and displayed so that the virtual object does not overlap with the real object. Wearable electronic device.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, If the above virtual object includes subtitles, convert the subtitles into speech and provide them to the user. Wearable electronic device.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, If the above virtual object contains financial information, the key points of the said financial information are summarized and provided to the user in text and / or voice, Wearable electronic device.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, If the above virtual object contains financial information, providing changes to the said financial information to the user in text and / or voice, Wearable electronic device.

10. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (410), the wearable electronic device, When the user moves a second distance in a second direction from the above reference point to a second position, Moving and positioning the above virtual object at a third location spaced a third distance in a third direction from the above reference point. Wearable electronic device.

11. In a method of operating a wearable electronic device (101, 400), Displays external real objects and virtual objects on a single screen on the display (250), and A virtual object is displayed at a first position spaced a first distance in a first direction from a reference point, and The local location of the wearable electronic device (101, 400) is set as a reference point, and If the user's gaze from the current location is in a direction where the virtual object cannot be seen, Providing information about the above virtual object to the user via voice, Method of operation of a wearable electronic device (101, 400).

12. In Paragraph 11, If the user and the virtual object are separated by more than a certain distance from the current location, or are close within a certain distance, information about the virtual object is provided to the user by voice. Method of operation of a wearable electronic device (101, 400).

13. In Paragraph 11, When the above virtual object overlaps with a real object, information about the above virtual object is provided to the user via voice. Method of operation of a wearable electronic device (101, 400).

14. In Paragraph 11, When the virtual object overlaps with a real object, the virtual object is moved and displayed so that the virtual object does not overlap with the real object. Method of operation of a wearable electronic device (101, 400).

15. In Paragraph 11, When the virtual object overlaps with a real object, the virtual object is separated into at least two parts and displayed so that the virtual object does not overlap with the real object. Method of operation of a wearable electronic device (101, 400).

Images