Electronic devices and methods of displaying virtual objects

Abstract

An electronic device according to one embodiment may comprise: at least one processor including a processing circuit; and a memory for storing instructions. The instructions may be individually or collectively executed by the at least one processor to instruct the electronic device to generate a second image in which a real space and a virtual space are composited, on the basis of a first image related to the real space captured through a camera. The instructions may be individually or collectively executed by the at least one processor to instruct the electronic device to generate, on the basis of a user input, a virtual object corresponding to a real object included in the first image. The instructions may be individually or collectively executed by the at least one processor to instruct the electronic device to determine a position at which the virtual object is to be displayed in the second image, on the basis of whether the real object is included within a specific physical range from the position of the user in the real space. The instructions may be individually or collectively executed by the at least one processor to instruct the electronic device to determine the size and depth of the virtual object on the basis of the content type of the virtual object. The instructions may be individually or collectively executed by the at least one processor to instruct the electronic device to display the virtual object in the second image according to a position at which the virtual object is to be displayed, and the size and depth of the virtual object. Various other embodiments may also be possible.

Description

Methods for displaying electronic devices and virtual objects

[0001] Various embodiments of the present invention relate to an electronic device and a method for displaying a virtual object.

[0002] Virtual reality (VR) refers to a specific environment or situation, or the technology itself, that is similar to reality but not real, created using artificial technology such as computers. Augmented reality (AR) is a branch of virtual reality (VR) and is a computer graphics technique that synthesizes virtual objects or information into an existing environment to make them appear as if they were present in the original environment. Recently, there has been a growing demand for technology that can apply real-time composite effects to images captured by cameras or render virtual objects that interact with the real environment in real time to provide simulated experiences in online environments. Accordingly, there is a need for the development of technology that can enhance user experience sharing by rendering virtual objects that interact naturally with the real environment and reflect changes in the real environment in real time.

[0003] Augmented reality technology is expanding its scope of application to include remote medical diagnosis, broadcasting, location-based services, mobile games, mobile solutions, and education.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0005] An electronic device according to one embodiment may include at least one processor including a processing circuit. The electronic device may include a memory storing instructions. The instructions may be individually or collectively executed by the at least one processor to cause the electronic device to generate a second image that composites a real space and a virtual space based on a first image of a real space captured by a camera. The instructions may be individually or collectively executed by the at least one processor to cause the electronic device to generate a virtual object corresponding to a real object included in the first image based on a user's input. The instructions may be individually or collectively executed by the at least one processor to cause the electronic device to determine a location at which to display the virtual object within the second image based on whether the real object is included within a specific physical range from a location of the user in the real space. The instructions may be individually or collectively executed by the at least one processor to cause the electronic device to determine a size and depth of the virtual object based on a content type of the virtual object. The instructions may be individually or collectively executed by the at least one processor to cause the electronic device to display the virtual object on the second image based on a location where the virtual object is to be displayed and the size and depth of the virtual object.

[0006] According to one embodiment, a method of operating an electronic device may include generating a second image that composites a real space and a virtual space based on a first image of a real space captured by a camera. The method may include generating a virtual object corresponding to a real object included in the first image based on a user's input. The method may include determining a location at which the virtual object is to be displayed in the second image based on whether the real object is included within a specific physical range from a location of the user in the real space. The method may include determining a size and a depth of the virtual object based on a content type of the virtual object. The method may include displaying the virtual object in the second image based on the location at which the virtual object is to be displayed and the size and depth of the virtual object.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2A illustrates an example of a perspective view of a wearable device according to one embodiment.

[0009] FIG. 2b illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.

[0010] FIGS. 3A and 3B illustrate an example of an appearance of a wearable device according to one embodiment.

[0011] FIG. 4 illustrates an example block diagram of a wearable device according to one embodiment.

[0012] FIG. 5 is a drawing for explaining a method of displaying a virtual object according to one embodiment.

[0013] FIG. 6 is an example of a flowchart of a method for displaying a virtual object according to one embodiment.

[0014] FIGS. 7A to 7F are drawings for explaining a method for creating a virtual object according to one embodiment.

[0015] FIG. 8 is a drawing for explaining a method for determining the position of a virtual object according to one embodiment.

[0016] FIG. 9 is a diagram illustrating a method for determining the size and depth of a virtual object according to one embodiment.

[0017] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0018]

[0019] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0020] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134).

[0021] According to one embodiment, the processor (120) may be implemented as a circuit (e.g., a processing circuit) such as a system on chip (SoC) or an integrated circuit (IC). The processor (120) may include one or more processors. For example, the processor (120) may include a combination of one or more processors such as a CPU, a GPU, an MPU, an AP, and a CP.

[0022] According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0023] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0024] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto.

[0025] According to one embodiment, the memory (130) may include one or more memories. Instructions stored in the memory (130) may be stored in a single memory. Instructions stored in the memory (130) may be divided and stored in multiple memories. Instructions stored in the memory (130) may be executed by a single processor (e.g., a main processor (121) or a secondary processor (123) such as a communication processor) or may be executed by multiple processors operating cooperatively (e.g., a main processor (121) and a secondary processor (123)).

[0026] According to one embodiment, the instructions stored in the memory (130) may be individually or collectively executed by the processor (120) to cause the electronic device (101) to perform and / or control the virtual object display method described with reference to FIGS. 5 to 9. The instructions stored in the memory (130) may be individually or collectively executed by a plurality of processors (e.g., the main processor (121) and / or the auxiliary processor (123)) to cause the electronic device (101) to perform and / or control the virtual object display method described with reference to FIGS. 5 to 9. According to one embodiment, the memory (130) may include a volatile memory (132) or a nonvolatile memory (134).

[0027] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0028] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0029] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0030] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0031] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0032] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0033] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0034] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0036] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0037] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0038] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0040] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0041] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0042] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0043] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0044] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0045]

[0046] FIG. 2A illustrates an example of a perspective view of a wearable device according to one embodiment. According to one embodiment, the wearable device (103) may have a form of glasses that can be worn on a body part (e.g., head) of a user. The wearable device (103) of FIGS. 2A and 2B may be an example of the electronic device (101) of FIG. 1. The wearable device (103) may include a head-mounted display (HMD). For example, the housing of the wearable device (103) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (103) may include one or more straps capable of being twined around the user's head, and / or one or more temples attachable to the ears of the head.

[0047] Referring to FIG. 2A, according to one embodiment, a wearable device (103) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0048] According to one embodiment, the wearable device (103) can be worn on a part of a user's body. The wearable device (103) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (103). For example, the wearable device (103) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition camera (or motion tracking camera) (260-2, 260-3) of FIG. 2B.

[0049] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0050] Referring to FIG. 2B, at least one display (250) can provide visual information transmitted from external light to a user through a lens included in at least one display (250), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of at least one display (250). When a user wears the wearable device (103), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0051] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (103) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0052] The wearable device (103) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (103) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (103) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (103) can view an image displayed on at least one display (250).

[0053] According to one embodiment, the frame (200) may be configured as a physical structure that allows the wearable device (103) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that, when the user wears the wearable device (103), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0054] Referring to FIG. 2A, the frame (200) may include a region (220) that is in contact with at least a portion of a user's body when the user wears the wearable device (103). For example, the region (220) of the frame (200) that is in contact with a portion of the user's body may include a region that is in contact with a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (103) makes contact with. According to one embodiment, the frame (200) may include a nose pad (210) that is in contact with a portion of the user's body. When the wearable device (103) is worn by the user, the nose pad (210) may be in contact with a portion of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another portion of the user's body that is distinct from the portion of the user's body.

[0055] For example, the frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (103) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200).

[0056] According to one embodiment, the wearable device (103) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (200).

[0057] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (103) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B, but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B. When the number of microphones (265) included in the wearable device (103) is two or more, the wearable device (103) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (200).

[0058] According to one embodiment, at least one optical device (282, 284) can project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) can be a projector. At least one optical device (282, 284) can be disposed adjacent to at least one display (250) or can be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (103) can include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0059] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (103). For example, the wearable device (103) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (103) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (103) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (103) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (103) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first area related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, or PPI (pixels per inch)) of a second area distinguished from the first area may be different from each other.The wearable device (103) can obtain an image (or screen) having a visual quality of a first area matching the user's gaze and a visual quality of a second area using foveated rendering. For example, if the wearable device (103) supports an iris recognition function, user authentication can be performed based on iris information obtained using a gaze tracking camera (260-1). An example in which the gaze tracking camera (260-1) is positioned toward both eyes of the user is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) can be positioned solely toward the user's left eye or right eye.

[0060] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (103) can compensate for depth information (e.g., the distance between the wearable device (103) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (260-4). The wearable device (103) can perform object recognition through an image acquired using a photographing camera (260-4). The wearable device (103) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the photographing camera (260-4). The wearable device (103) can perform a pass-through function to display an image acquired through the photographing camera (260-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (250). The photographing camera (260-4) may be referred to as a high resolution (HR) camera or a photo video (PV) camera.The camera (260-4) may provide an auto focus (AF) function and an optical image stabilization (OIS) function. The camera (260-4) may include a global shutter (GS) camera and / or a rolling shutter (RS) camera. In one embodiment, the camera (260-4) may be positioned on a bridge (203) positioned between the first rim (201) and the second rim (202).

[0061] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (103) and thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (103) looks straight ahead, the wearable device (103) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (103) is positioned.

[0062] The motion recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The motion recognition camera (260-2, 260-3) can recognize the user's motion (gesture recognition), obtain a signal corresponding to the motion, and provide a display corresponding to the signal on at least one display (250). A processor (e.g., the processor (120) of FIG. 1) can identify the signal corresponding to the motion, and perform a designated function based on the identification. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 DOF pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) may be disposed on the first limb (201) and / or the second limb (202). The motion recognition cameras (260-2, 260-3) may include a global shutter (GS) camera (e.g., a global shutter (GS) camera) used for head tracking, hand tracking, and / or spatial recognition based on one of a three-degree-of-freedom pose or a six-degree-of-freedom pose. The GS camera may include two or more stereo cameras to track fine movements. As an example, the GS camera may be included in the gaze tracking camera (260-1) for tracking the gaze of a user.

[0063] The camera (260) included in the wearable device (103) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (103) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (103) can identify an external object based on a sensor for identifying the distance between the wearable device (103) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (103) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (103).

[0064] Although not shown, in one embodiment, the wearable device (103) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (200) and the hinge units (206, 207).

[0065] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (103). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0066] The antenna module (275) can transmit signals or power to the outside of the wearable device (103), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0067] The speaker (255) can output an audio signal to the outside of the wearable device (103). The audio output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (103). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus positioned adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus positioned adjacent to the user's right ear.

[0068] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state, in order to visually provide information regarding a specific state of the wearable device (103) to the user. For example, when the wearable device (103) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0069] Referring to FIG. 2B, according to one embodiment, a wearable device (103) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (103) (e.g., hardwares illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (103) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0070] According to one embodiment, the wearable device (103) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (103) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (103). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (103) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (103) based on the IMU.

[0071]

[0072] Figures 3a and 3b illustrate an example of an exterior appearance of a wearable device according to one embodiment. The wearable device (103) of Figures 3a and 3b may be an example of the wearable device (103) of Figure 1. An example of an exterior appearance of a first side (310) of a housing of the wearable device (103) according to one embodiment is illustrated in Figure 3a, and an example of an exterior appearance of a second side (320) opposite to the first side (310) may be illustrated in Figure 3b.

[0073] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (103) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (103) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (103) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0074] According to one embodiment, the wearable device (103) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (103) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (103) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (103) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar expressing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (103).

[0075] Referring to FIG. 3b, a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor (330)) for obtaining information related to the external environment of the wearable device (103) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3a. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0076] For example, using cameras (260-11, 260-12), the wearable device (103) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be placed on the second face (320) of the wearable device (103) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be placed on the second face (320) of the wearable device (103) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. As an example, the wearable device (103) can obtain a single screen using a plurality of images obtained through the cameras (260-11, 260-12). Cameras (260-11, 260-12) may be referred to as the shooting camera (260-4) of FIG. 2b.

[0077] According to one embodiment, the wearable device (103) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (103) and an external object. Using the depth sensor (330), the wearable device (103) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (103). Although not illustrated, a microphone may be disposed on the second face (320) of the wearable device (103) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0078]

[0079] FIG. 4 illustrates an example of a block diagram of a wearable device according to one embodiment. Referring to FIG. 4, a wearable device (103) according to one embodiment may include at least one of a processor (410) (e.g., the processor (120) of FIG. 1), a memory (415) (e.g., the memory (130) of FIG. 1), a display (420) (e.g., the display module (160) of FIG. 1), a camera (425) (e.g., the camera module (180) of FIG. 1), a sensor (430) (e.g., the sensor module (176) of FIG. 1), or a communication circuit (435) (e.g., the communication module (190) of FIG. 1). The processor (410), the memory (415), the display (420), the camera (425), the sensor (430), and the communication circuit (435) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (402). The type and / or number of hardware components included in the wearable device (103) are not limited to those illustrated in FIG. 4. For example, the wearable device (103) may include only some of the hardware components illustrated in FIG. 4. Elements within the memory described below (e.g., layers and / or modules) may be logically separated. Elements within the memory (415) may be included within a hardware component that is distinct from the memory (415). The operation performed by the processor (410) using each element within the memory (415) is one embodiment, and the processor (410) may perform a different operation from the above operation through at least one element among the elements within the memory (415).

[0080] The processor (410) of the wearable device (103) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (410) may be one or more. For example, the processor (410) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0081] According to one embodiment, the processor (410) may be implemented as a circuit (e.g., a processing circuit) such as a system on chip (SoC) or an integrated circuit (IC). The processor (410) may include one or more processors. For example, the processor (410) may include a combination of one or more processors such as a CPU, a GPU, an MPU, an AP, and a CP.

[0082] The memory (415) of the wearable device (103) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (410).

[0083] According to one embodiment, the memory (415) may include one or more memories. Instructions stored in the memory (415) may be stored in a single memory. Instructions stored in the memory (415) may be divided and stored in multiple memories. Instructions stored in the memory (415) may be executed by a single processor (e.g., a main processor or a secondary processor such as a communication processor) or by multiple processors operating cooperatively (e.g., a main processor and a secondary processor).

[0084] According to one embodiment, the instructions stored in the memory (415) may be individually or collectively executed by the processor (410) to cause the wearable device (103) to perform and / or control the virtual object display method described with reference to FIGS. 5 to 9. The instructions stored in the memory (415) may be individually or collectively executed by a plurality of processors (e.g., a main processor and / or a secondary processor) to cause the wearable device (103) to perform and / or control the virtual object display method described with reference to FIGS. 5 to 9.

[0085] According to one embodiment, the memory (415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC).

[0086] In one embodiment, the display (420) of the wearable device (103) can output visualized information to a user of the wearable device (103). For example, the display (420) can be controlled by a processor (410) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (420) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0087] In one embodiment, the camera (425) of the wearable device (103) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (425) may be arranged in the form of a two-dimensional array. The camera (425) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (425) may mean one (a) two-dimensional frame data acquired from the camera (425). For example, video data captured using the camera (425) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (425) according to a frame rate. The camera (425) may further include a flash light that is positioned toward the direction in which the camera (425) receives light and outputs light toward the direction.

[0088] According to one embodiment, the wearable device (103) may include a plurality of cameras, for example, cameras (425), arranged in different directions. A first camera among the plurality of cameras may be referred to as a motion recognition camera (e.g., motion recognition cameras 260-2 and 260-3 of FIG. 2B ), and a second camera may be referred to as a gaze tracking camera (e.g., gaze tracking camera 260-1 of FIG. 2B ). The wearable device (103) may identify a position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (103) may identify a direction of a gaze of a user wearing the wearable device (103) using an image acquired using the second camera. For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0089] According to one embodiment, a sensor (430) of a wearable device (103) may generate electrical information that may be processed by a processor (410) and / or a memory (415) of the wearable device (103) from non-electronic information related to the wearable device (103). The information may be referred to as sensor data. The sensor (430) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (103), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (103).

[0090] In one embodiment, the communication circuit (435) of the wearable device (103) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (103) and an external electronic device. The communication circuit (435) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (435) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0091] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (410) of the wearable device (103) may be stored in the memory (415) of the wearable device (103). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (103) and / or the processor (410) may perform at least one of the operations of FIG. 6 or FIG. 11 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (103) may mean that one or more instructions provided in the form of an application are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (103)). For example, an application may include a program and / or a library related to a service provided to a user.

[0092] Referring to FIG. 4, programs installed in the wearable device (103) may be classified into any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (103) (e.g., the display (420), the camera (420), and / or the sensor (430)) may be classified within the hardware abstraction layer (480). The framework layer (450) may be referred to as an XR framework layer in that it includes one or more programs for providing an XR (extended reality) service. For example, FIG. 4 illustrates layers being divided within the memory (415), the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (415).

[0093] For example, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a position tracker (471), a space recognizer (472), a gesture tracker (473), and / or an eye tracker (474), a face tracker (475)) may be classified. Programs classified within the framework layer (450) may provide an executable API (application programming interface) based on other programs.

[0094] For example, within the application layer (440), programs designed to target users controlling wearable devices (103) may be classified. Examples of programs classified into the application layer (440) include, but are not limited to, an XR (extended reality) system UI (user interface) and / or an XR application (442). For example, programs (e.g., software applications) classified into the application layer (440) may call an API (application programming interface) to cause execution of functions supported by programs classified into the framework layer (450).

[0095] For example, the wearable device (103) may display one or more visual objects on the display (420) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (441). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (103) may provide a service for controlling functions available within a virtual space to the user based on the execution of the XR system UI (441).

[0096] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plug-in (444) are illustrated as being included within the XR system UI (441), but are not limited thereto. For example, the XR system UI (441) may cause execution of functions supported by the lightweight renderer (443) and / or the XR plug-in (444) included within the application layer (440).

[0097] For example, the wearable device (103) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (103) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (444). The XR plugin (444) may be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0098] For example, the wearable device (103) may display a screen representing at least a portion of a virtual space on the display (420) based on the execution of the XR application (442). The XR plug-in (444-1) included in the XR application (442) may be referenced by the XR plug-in (444) of the XR system UI (441). Descriptions of the XR plug-in (444-1) that overlap with the description of the XR plug-in (444) may be omitted. The wearable device (103) may cause the execution of the screen composition manager (451) based on the execution of the XR application (442).

[0099] For example, the wearable device (103) can convert a screen representing at least a portion of a virtual space into a three-dimensional screen and display it on the display (420). The wearable device (103) can convert an application (or object) without 3D information into an application (or object) having 3D information (e.g., information about binocular disparity (e.g., depth information reflected)) through Space Flinger. According to one embodiment, the wearable device (103) can provide a virtual space service based on the execution of the screen configuration manager (451). For example, the screen configuration manager (451) can include a platform (e.g., an Android platform) for supporting the virtual space service. The wearable device (103) can display, on the display, the posture of a virtual object representing the user's posture rendered using data acquired through the sensor (430) based on the execution of the screen configuration manager (451). The screen composition manager (451) may be referred to as a CPM (composition presentation manager).

[0100] For example, the screen composition manager (451) may include a runtime service (452). As an example, the runtime service (452) may be referred to as an OpenXR runtime module. The wearable device (103) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (103) based on the execution of the runtime service (452). As an example, the wearable device (103) may be used to perform rendering for a virtual space service to a user based on the execution of the runtime service (452). For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the execution of the runtime service (452).

[0101] For example, the screen configuration manager (451) may include a pass-through library (453). Based on the execution of the pass-through library (453), the wearable device (103) may display a screen representing a virtual space on the display (420), while another screen representing a real space acquired through the camera (425) may be superimposed on at least a portion of the screen.

[0102] For example, the screen composition manager (451) may include a renderer (e.g., the renderer (540-1) of FIG. 5). The wearable device (103) may render a screen to be displayed on the display by compositing virtual layers (or virtual nodes) rendered based on sensor data (e.g., sensing data acquired through the camera (425) or sensor (430)) and pass-through layers (or pass-through nodes) acquired through the pass-through library (453) using the screen composition manager (451). The virtual layers may be referred to as virtual nodes and / or virtual surfaces. The wearable device (103) may render each of the virtual layers or all of the virtual layers through the screen composition manager (451).

[0103] For example, the screen configuration manager (451) may include an input manager (454). Based on the execution of the input manager (454), the wearable device (103) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (470). The wearable device (103) may initiate execution of at least one of the functions of the wearable device (103) using the acquired data.

[0104] For example, the perception abstract layer (460) can be used for data exchange between the screen configuration manager (451) and the perception service layer (470). From the perspective of being used for data exchange between the screen configuration manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referred to as OpenPX and / or PPAL (perception platform abstract layer). The perception abstract layer (460) can be used for a perception client and a perception service.

[0105] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data acquired from a sensor (430) (or a camera (425)). The one or more programs may include at least one of a position tracker (471), a space recognizer (472), a gesture tracker (473), an eye tracker (474), and / or a face tracker (475). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those illustrated in FIG. 4.

[0106] For example, the wearable device (103) can identify the pose of the wearable device (103) using the sensor (430) based on the execution of the position tracker (471). The wearable device (103) can identify the 6 degrees of freedom pose (6 DOF pose) of the wearable device (103) using data acquired using the camera (425) and the IMU based on the execution of the position tracker (471). The position tracker (471) may be referred to as a head tracking (HeT) module.

[0107] For example, the wearable device (103) may be used to construct a three-dimensional virtual space around the wearable device (103) (or a user of the wearable device (103)) based on the execution of the space recognizer (472). Object / object recognition information may be acquired by the execution of the space recognizer (472). The object / object recognition information may include information for classifying objects included in an image based on information acquired from the first camera and / or the second camera and recognizing each classified object. The classified objects may be objects, people, backgrounds, etc. The wearable device (103) may reconstruct a three-dimensional surrounding environment of the wearable device (103) using data acquired using the camera (425) based on the execution of the space recognizer (472). The wearable device (103) can identify at least one of a plane, a slope, or stairs based on the surrounding environment of the wearable device (103) reproduced in three dimensions based on the execution of the spatial recognizer (472). The spatial recognizer (472) may be referred to as a scene understanding (SU) module.

[0108] For example, the wearable device (103) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (103) based on the execution of the gesture tracker (473). Gesture tracking (Hand Tracking) information may be acquired by the gesture tracker (473). The gesture tracking information may include information acquired from a first camera (e.g., the motion recognition cameras (260-2, 260-3) of FIG. 2B) and information acquired by tracking a hand pose of the user using ToF information. The wearable device (103) may recognize a gesture based on the hand pose. For example, the wearable device (103) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (430) based on the execution of the gesture tracker (473). For example, the wearable device (103) may identify a pose and / or gesture of a user's hand based on data (or images) acquired using a camera (425) based on the execution of a gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0109] For example, the wearable device (103) can identify (or track) eye movements of a user of the wearable device (103) based on the execution of the eye tracker (474). Eye tracking information can be obtained by the execution of the eye tracker (474). The eye tracking information can be obtained by estimating eye movements (e.g., pupil movements) of the user using an eye tracking camera (e.g., eye tracking camera (260-1) of FIG. 2A) and an IR LED. The wearable device (103) can perform eye tracking based on the eye movement estimation information. For example, the wearable device (103) can identify eye movements of the user using data acquired from at least one sensor based on the execution of the eye tracker (474). For example, the wearable device (103) may identify the movement of a user's eyes based on data acquired using a camera (425) (e.g., the eye tracking camera (260-1) of FIGS. 2A and 2B) and / or an infrared light emitting diode (IR LED) based on the execution of an eye tracker (474). The eye tracker (474) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0110] For example, the recognition service layer (470) of the wearable device (103) may further include a face tracker (475) for tracking the user's face. For example, the wearable device (103) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (475). The wearable device (103) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (475). As an example, the wearable device (103) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (475).

[0111]

[0112] FIG. 5 is a drawing for explaining a method of displaying a virtual object according to one embodiment.

[0113] Referring to FIG. 5, according to one embodiment, an electronic device (500) (e.g., the electronic device (101) of FIG. 1 and / or the wearable device (103) of FIG. 2A) may include an object detector (505), a surface detector (510), an image analyzer (515), an optical character recognition (OCR) module (520), a text analyzer (525), a cropping module (530), a generating module (or an artificial intelligence (AI) module) (535), a 3D rendering module (540), and a presentation module (545).

[0114] According to one embodiment, the object detector (505), the surface detector (510), the image analyzer (515), the OCR module (520), the text analyzer (525), the cropping module (530), the generative module (535), the 3D rendering module (540), and the display module (545) may be implemented as one processor (e.g., the processor (120) of FIG. 1 and / or the processor (410) of FIG. 4) or multiple processors (e.g., the processor (120) of FIG. 1 and / or the processor (410) of FIG. 4).

[0115] According to one embodiment, the object detector (505) can detect a real object included in a first image of a real space. The real space may refer to a space in the real world where a user wearing an electronic device (or wearable device) (101) (e.g., an augmented reality (AR) and / or a video see-through device (VST)) exists. The real object may include a physical object that can be displayed as a virtual object through object recognition (and / or object recognition) in a real environment displayed to a user wearing the electronic device (101). The first image may include an image of a real environment acquired through a pass-through library (e.g., the pass-through library (453) of FIG. 4). The first image may be captured through a camera included in the electronic device (101) (e.g., the camera module (180) of FIG. 1 and the camera (425) of FIG. 4). For example, the object detector (505) can detect a real object included in the first image through an object recognition method.

[0116] According to one embodiment, the surface detector (510) can identify a planar area of ​​a real object. The surface detector (510) can identify a planar area of ​​a real object detected by the object detector (505). For example, the real object may be a three-dimensional object (e.g., a hexahedron). The surface detector (510) can identify a planar area of ​​one of the hexahedrons. The surface detector (510) can identify a planar area that contains the most various elements (e.g., images and text) among the multiple planar areas of the hexahedron. In the above example, the real object is described as a hexahedron, but is not limited thereto, and the real object may be a complex shape such as a polyhedron such as a sphere, a tetrahedron, an octahedron, or a combination thereof. The planar area may also have a shape of various geometric figures such as a rectangle, a square, a polygon, and / or a circle. However, the planar region is not limited to the shape of a geometric figure, and may also include a shape that substantially corresponds to a geometric figure. For example, if a portion of a real-world object (e.g., a hexahedron) includes a shape other than a circle or a perfect square, the surface detector (510) may identify a portion of the real-world object as a planar region within a certain tolerance range.

[0117] According to one embodiment, the cropping module (530) may crop a flat area (e.g., an identified area corresponding to a user input) from a first image (e.g., an image of a real environment acquired through a pass-through library (453)) captured by a camera (e.g., a camera (425) of FIG. 4). The cropping module (530) may generate a cropped image by cropping the identified area corresponding to the user input. The cropping module (530) may obtain information about the flat area detected by the surface detector (510). The cropped image may be generated to improve the image quality of the virtual object. The cropped image may be transmitted from the cropping module (530) to a generative model (535), and up-scaling may be performed through the generative model (535). However, if text is included in the cropped image, the cropped image may be preprocessed by an image analyzer (515), an OCR module (520), and a text analyzer (525) to improve the quality of the text. Hereinafter, the operations of the image analyzer (515), the OCR module (520), and the text analyzer (525) to improve the quality of the text in the cropped image will be described in detail.

[0118] In one embodiment, the image analyzer (515) may obtain (e.g., receive) a cropped image from the cropping module (535) and identify whether text is included in the cropped image. The text may include various languages, such as Korean, English, Japanese, and Chinese. For example, if text is included in the cropped image, the image analyzer (515) may output the cropped image to the OCR module (520) to extract the text.

[0119] According to one embodiment, the OCR module (520) may acquire (e.g., receive) a cropped image determined by the image analyzer (515) to contain text. The OCR module (520) may identify and extract text within the cropped image through OCR. For example, the OCR module (520) may extract text within the cropped image by comparing the cropped image with a font of a pre-stored text (e.g., stored in a memory (e.g., memory (130) of FIG. 1 and / or memory (415) of FIG. 4) and / or a cache (e.g., a cache within the OCR module (520)).

[0120] According to one embodiment, the text analyzer (525) can analyze the extracted text to analyze the language, font, size, and / or color of the text. The text analyzer (525) may exist as a separate module, as illustrated in FIG. 5, or may be included in the OCR module (525). For example, the OCR module (520) and the text analyzer (525) may be implemented as separate processors or as a single processor.

[0121] According to one embodiment, the generative module (535) may be implemented using various artificial intelligence technologies such as an artificial intelligence neural network and machine learning that generate new data (e.g., virtual objects) based on an input (e.g., a cropped image). The generative module (535) may be an image generation module. The image generation module may include generative artificial intelligence (AI). The image generation module may identify various types of inputs, such as images and / or videos, and generate new data corresponding to the inputs. The image generation module may be a model trained to generate the most appropriate statistical output based on the inputs.

[0122] According to one embodiment, the generation module (535) may be built into the electronic device (500) as illustrated in FIG. 5, but is not limited thereto. For example, the generation module (535) may be implemented as a server (not illustrated) external to the electronic device (500), so that the operations of the generation module (535) may be performed by the external server.

[0123] In one embodiment, the generative module (535) can transform a planar area into a virtual object based on a cropped image. For example, the generative module (535) can be trained to generate a statistically most appropriate virtual object based on the cropped image.

[0124] According to one embodiment, the generative module (535) may first perform upscaling of a cropped image to improve the image quality of a virtual object. The upscaling may be performed through an upscaling method of a Convolutional Neural Network (CNN), a Super Resolution Convolutional Neural Network (SRCNN), and / or a GAN-based Super-Resolution Generative Adversarial Network (SRGAN). The generative module (535) may generate a virtual object that provides an ultra-high resolution image quality. This may prevent the image quality of the virtual object from deteriorating even when the virtual object is later enlarged according to a user input. For example, an image captured by a camera that provides a wide field of view (or FOV) (e.g., a wide camera and / or an ultrawide camera) may have low clarity. The generative module (535) may improve the clarity of an image captured by a camera that provides a wide field of view (or FOV) through an upscaling method of a GAN-based SRGAN. This allows virtual objects with ultra-high resolution image quality to be created.

[0125] According to one embodiment, the generative module (535) may upscale the cropped image using the upscaling method described above if the cropped image does not contain text. However, if the cropped image contains text, the generative module (535) may additionally upscale only the text in the cropped image based on information obtained by the image analyzer (515), the OCR module (520), and / or the text analyzer (525) (e.g., the text contained in the cropped image and the analysis result of the text (e.g., the font, size, and / or color of the text)). As a result, the text in the cropped image is expressed more clearly, and since the text is recognized as text itself in the virtual object, actions (e.g., selection and / or copy) according to additional user input (e.g., selection request and / or copy request, etc.) may be performed on the text itself. For example, even if text is included in a cropped image, if separate text processing (e.g., operations performed by the image analyzer (515), the OCR module (520), and the text analyzer (525)) is not performed, the text is recognized together with the image, not the text itself, and processing (e.g., selection and / or copying) for the text itself cannot be performed. That is, the image analyzer (515), the OCR module (520), the text analyzer (525), and the generative module (535) can provide additional services for the text itself (e.g., performing operations according to the user's additional input for the text itself in a virtual space) by processing differently the cases where text is included in the cropped image and the cases where it is not included.

[0126] According to one embodiment, the 3D rendering module (540) may obtain (e.g., receive) a virtual object from the generation module (535) and render the virtual object in 3D. The 3D rendering module (540) may operate in a spatial layer and / or a screen composition layer.

[0127] According to one embodiment, the display module (545) can generate a second image of the virtual space based on the rendered virtual object obtained from the 3D rendering module (540) and the first image (e.g., an image of the real space) (e.g., an image of the real environment obtained through the pass-through library (453). The display module (545) can generate the second image by aligning the first image and the rendered virtual object. The display module (545) can determine the position, size, and / or depth of the virtual object to be displayed in the second image. That is, the object detector (505), the surface detector (510), the image analyzer (515), and the OCR module (520) described above. The operations performed by the text analyzer (525), the cropping module (530), the generative module (535), and the 3D rendering module (540) may relate to a method of generating a virtual object (e.g., converting a real object included in a first image into a virtual object), and the operations performed by the display module (545) may relate to a method of displaying a virtual object in a virtual space to provide a second image. Hereinafter, a method of generating and displaying a virtual object will be described in more detail with reference to FIGS. 6 to 9.

[0128]

[0129] FIG. 6 is an example of a flowchart of a method for displaying a virtual object according to one embodiment.

[0130] Referring to FIG. 6, according to one embodiment, operations 610 to 690 may be performed sequentially, but are not necessarily performed sequentially. For example, the order of each operation (610 to 690) may be changed, and at least two operations may be performed in parallel.

[0131] In operation 610, an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable device (103) of FIG. 2A, and / or the electronic device (500) of FIG. 5) may generate a second image based on a first image of a real space captured by a camera (e.g., the camera module (180) of FIG. 1 and / or the camera (425) of FIG. 4)) (e.g., an image of a real environment acquired through a pass-through library (e.g., the pass-through library (453) of FIG. 4)). The second image may be an image in which a real space and a virtual space are composited. The virtual space may mean a space in which digital elements (e.g., virtual objects) are added to a real space. In the second image, virtual information may be overlapped with the real space, so that the boundary between reality and virtuality may be expressed vaguely. The second image may represent a space in which the virtual space and the real space are composited. That is, in the second image, various real objects can be displayed together with virtual objects, and through operations 630 to 690, one or more of the various real objects can be converted into virtual objects and displayed.

[0132] According to one embodiment, the electronic device (500) can detect an object that can be displayed as a virtual object included in a first image and identify a planar area of ​​the detected object. The electronic device (500) can provide (or output) an interface to a user of the electronic device (500) that proposes converting the identified planar area into a virtual object. The user can request the electronic device (500) to convert a planar area corresponding to the interface into a virtual object through the interface.

[0133] In operation 630, the electronic device (500) may generate a virtual object corresponding to a real object included in the first image based on a user's input. The user's input may be obtained through an interface displayed in the second image (e.g., an interface proposing a conversion of a real object (e.g., a planar area of ​​the real object) into a virtual object). For example, the electronic device (500) may provide the user with an interface (e.g., an interface proposing conversion of one of various real objects into a virtual object) through the second image. When at least one real object is selected by the user through the interface, the electronic device (500) may generate a virtual object corresponding to the specific real object. A specific method of generating a virtual object will be described in more detail with reference to FIGS. 7A to 7F.

[0134] In operation 650, the electronic device (500) may determine a location for displaying a virtual object within the second image based on whether the real object is within a specific physical range from the user's location in real space. The specific physical range may correspond to the user's action radius. The user's action radius may be within a certain range based on the direction in which the user's upper body is facing, and may include a physical range within the user's reach. A method for determining the location of the virtual object will be described in more detail with reference to FIG. 8.

[0135] In operation 670, the electronic device (500) may determine the size and depth of the virtual object based on the content type of the virtual object. The content type may include an image-related type and / or a text-related type. However, the types of content types are not limited thereto, and other types besides images and text may also exist.

[0136] According to one embodiment, the electronic device (500) can identify the content type of the virtual object. For example, the electronic device (500) can determine the ratio of the image and the ratio of the text included in the virtual object. The electronic device (500) can compare the ratio of the image and the ratio of the text included in the virtual object. If the ratio of the image is large, the electronic device (500) can determine the content type of the virtual object as an image-related type. If the ratio of the text is large, the electronic device (500) can determine the content type of the virtual object as a text-related type.

[0137] According to one embodiment, the electronic device (500) may display a virtual object to better present the content, depending on the content type of the virtual object. A method for determining the size and depth of a virtual object will be described in more detail with reference to FIG. 9.

[0138] In operation 690, the electronic device (500) may display a virtual object on the second image according to the location at which the virtual object is to be displayed, the size, and the depth of the virtual object. The electronic device (500) may display the virtual object on the second image according to the location, size, and depth of the virtual object determined through operations 610 to 670. However, if there is past history information on when the virtual object was displayed, the electronic device (500) may preferentially determine the location, size, and / or depth of the virtual object based on the past history information. The past history information may well reflect the preferences (or tastes) of the user of the electronic device (500).

[0139] According to one embodiment, the electronic device (500) may display a virtual object in response to additional input from a user (e.g., a request to change the position, size, and / or depth of the virtual object).

[0140]

[0141] FIGS. 7A to 7F are drawings for explaining a method for creating a virtual object according to one embodiment.

[0142] Referring to FIG. 7A, according to one embodiment, operations 705 to 720 may be performed sequentially, but are not necessarily performed sequentially. For example, the order of each operation (705 to 720) may be changed, and at least two operations may be performed in parallel.

[0143] In operation 705, an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable device (103) of FIG. 2A, and / or the electronic device (500) of FIG. 5) (e.g., the object detector (505) of FIG. 5) may detect an object that can be displayed as a virtual object in a real space displayed to a user wearing the electronic device (500). The object that can be displayed as a virtual object may be an object that includes one or more planar areas to create a planar area as a virtual object. The planar area may refer to a surface of the object. The surface of the object may be in the form of a recognizable two-dimensional surface, and may be in the form of a plane and / or a curved surface. For example, the object that can be displayed as a virtual object may be a hexahedron, as illustrated in FIG. 7B. However, the present invention is not limited thereto, and the object that can be displayed as a virtual object may include a sphere or various undefined shapes.

[0144] Referring to FIG. 7B, according to one embodiment, the electronic device (500) can detect a real object (735) including a planar area (e.g., a surface) among a plurality of real objects included in a first image (730) of a real space. The detected real object (735) is a three-dimensional object and can include a plurality of planar areas.

[0145] In operation 710, the electronic device (500) (e.g., the surface detector (510) of FIG. 5) can identify a planar area of ​​the detected real object. The planar area can have various shapes, such as a rectangle, a square, a polygon, a circle, etc. The planar area can include an image, text, and / or a combination of the two. For example, the planar area can include a pattern (e.g., including various patterns), a diagram, and / or a graph. The electronic device (500) (e.g., the cropping module (530) of FIG. 5) can crop the planar area (e.g., the identified area corresponding to the user input). The electronic device (500) (e.g., the image analyzer (515) of FIG. 5) can identify whether text is included in the cropped image. If text is included in the cropped image, the electronic device (500) can extract the text through an OCR technique (e.g., performed by the OCR module (520) of FIG. 5). An electronic device (500) (e.g., a text analyzer (525) of FIG. 5) can analyze text extracted through an OCR technique to analyze the language, font, size, and / or color of the text included in a flat area.

[0146] In one embodiment, if the flat area of ​​the detected object has a plain, monochromatic pattern without any pattern or text, the flat area may not be identified (or detected). This may be because there is no content to be provided to the user on the flat area of ​​the virtual object. In other words, the electronic device (500) can identify only the flat area with content and provide it to the user as a virtual object.

[0147] Referring to FIG. 7C, according to one embodiment, the electronic device (500) can identify a planar area of ​​a detected real object (735). At this time, since neither an image nor text is engraved on the side of the real object (735), there may be no content to be provided through the side of the real object (735). On the other hand, since an image (e.g., a bear image and an eye image) and text (e.g., “MERRY CHRISTMAS”) are engraved on the top surface of the real object (735), there may be content to be provided through the top surface of the real object (735). The electronic device (500) can identify only a planar area (740) (e.g., the top surface) of the detected real object (735) that has content.

[0148] In operation 715, the electronic device (500) may display an interface indicating that the identified area can be displayed as a virtual object (e.g., an interface suggesting conversion of the identified planar area (740) into a virtual object).

[0149] Referring to FIG. 7D , according to one embodiment, the interface (745) may be displayed as a separate icon at a vertex of a planar area. However, the present invention is not limited thereto, and the interface (745) may be displayed with variations in color, brightness, size, etc., or may be displayed as an icon, image, message, etc. different from the icon illustrated in FIG. 7 . In addition, the interface may be within or around the identified planar area. The interface may also be displayed at a fixed location on a separate second image unrelated to the identified planar area.

[0150] In operation 720, the electronic device (500) (e.g., the cropping module (530) and / or the generating module (535) of FIG. 5) may generate the identified planar area (740) as a virtual object based on a user input via the interface (745). When the user selects the planar area (740) via the interface (745), the electronic device (500) (e.g., the cropping module (530) of FIG. 5) may crop the planar area (740) to generate a cropped image (750). The electronic device (500) (e.g., the generating module (535) of FIG. 5) may process the cropped image (750) to generate it as a virtual object.

[0151] Referring to FIG. 7E, according to one embodiment, the electronic device (500) may obtain a user input (e.g., a request to convert a planar area (740) into a virtual object in response to an interface (745). The electronic device (500) may crop the planar area (750) according to the user input to generate a crop image (750). The electronic device (500) may input the crop image (750) to a generation module (e.g., the generation module (535) of FIG. 5) and, before converting it into a virtual object, may preprocess the crop image (750) to improve the image quality of the virtual object. The crop image (750) may be preprocessed differently based on whether it contains text.

[0152] In one embodiment, if the cropped image does not contain text, the electronic device (500) may upscale the entire cropped image. The electronic device (500) may then generate a virtual object using the upscaled cropped image. The virtual object thus generated may be recognized as a single image and displayed in the second image.

[0153] According to one embodiment, if the cropped image (750) includes text, the electronic device (500) may extract and / or analyze text information included in the cropped image (750). For example, the electronic device (500) may extract text (e.g., "MERRY CHRISTMAS") within the cropped image (750) through OCR. The electronic device (500) may analyze the extracted text to analyze the language, font, size, and / or color of the text. Based on the text information, the electronic device (500) may further upscale the text area within the cropped image (750). Further upscaling the text area within the cropped image (750) makes the image quality of the text area clearer than other areas, thereby displaying the text more clearly, and since the text area included in the virtual object is recognized as text itself, only the text area, rather than the entire area of ​​the virtual object, may be selected, copied, or otherwise processed. That is, a virtual object created in this way can be displayed on a second image by recognizing the image area and text area separately, rather than as a single image as a whole. If the text and image are recognized as a single image (e.g., if upscaling is not performed on the separate text area), there may be a problem in that operations such as selecting and / or copying only the text included in the image cannot be performed. On the other hand, if the image area and text area are recognized separately through upscaling on the separate text area, the user can perform operations such as selecting and / or copying only the text included in the image.

[0154] Referring to FIG. 7F, according to one embodiment, the electronic device (500) may change the text included in the cropped image (750). If the cropped image (750) includes text, the electronic device (500) extracts and / or analyzes the text information included in the cropped image (750) for upscaling, and may provide various services based on the text information. For example, the electronic device (500) may change the language of existing text (e.g., “MERRY CHRISTMAS”) and provide it as text in a different language (e.g., “Merry Christmas”). For example, if the system language of the electronic device (500) (e.g., the language set when the electronic device (500) is initially booted) and the language of the analyzed text are different, the electronic device (500) may change the language of the analyzed text to the system language and provide it. In addition, the electronic device (500) may change the font, size, and / or color of the existing text and provide it.

[0155] According to one embodiment, a virtual object may be generated based on a planar area and displayed as a two-dimensional object in the second image. However, the present invention is not limited thereto, and the virtual object may also be displayed as a three-dimensional object. When the virtual object is displayed as a three-dimensional object, if the electronic device (500) lacks the information necessary to convert the two-dimensional virtual object into a three-dimensional object, data (e.g., information necessary to convert the two-dimensional virtual object into a three-dimensional object) may be acquired from a camera of another electronic device (not shown) connected to the vicinity (e.g., another electronic device connected to the vicinity of the electronic device (500)) (e.g., a camera mounted in a fixed location) and used for three-dimensional object conversion. Alternatively, the generation module (535) may display the virtual object through a three-dimensional image inferred based on the current image.

[0156] Above, a method for creating a virtual object has been described in detail. Below, a method for displaying a virtual object in a second image will be described in detail with reference to FIGS. 8 and 9.

[0157]

[0158] FIG. 8 is a drawing for explaining a method for determining the position of a virtual object according to one embodiment.

[0159] Referring to FIG. 8, according to one embodiment, operations 810 to 870 may be performed sequentially, but are not necessarily performed sequentially. For example, the order of each operation (810 to 870) may be changed, and at least two operations may be performed in parallel.

[0160] In operation 810, an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable device (103) of FIG. 2A, and / or the electronic device (500) of FIG. 5) may generate an identified plane area as a virtual object based on a user input via an interface (e.g., the interface (745) of FIG. 7D). Operation 810 is substantially the same as operation 720 of FIG. 7, and thus, redundant descriptions will be omitted.

[0161] In operation 820, the electronic device (500) may determine whether the real object includes a display of the electronic device. This may be for generating a virtual object through a method other than image processing (e.g., image processing according to operations 705 to 720 of FIG. 7 ) if the real object includes a display of the electronic device. If the real object does not include a display of the electronic device, the electronic device (500) may perform operation 850. If the real object includes a display of the electronic device, the electronic device (500) may perform operation 830.

[0162] According to one embodiment, the electronic device (500) can determine whether the real object is an electronic device (e.g., a notebook, a desktop, and / or a smartphone). If the real object includes a notebook screen, the electronic device (500) can mirror a flat area (e.g., the notebook screen) to create a virtual object. This will be described in detail through operations 830 and 840.

[0163] In operation 830, the electronic device (500) can determine whether it is connected to an electronic device of a real object. The electronic device (500) can determine whether there is a wireless and / or wired connection between the electronic device (500) and the real object. The wired connection can include a wired connection based on peripheral component interconnect express (PCIe), fiber optic distribution data interface (FDDI), digital subscriber line (DSL), Ethernet, asynchronous transfer mode (ATM), frame relay, data over cable service interface specification (DOCSIS), or any other wired communication protocol. Wireless connectivity includes RF, IR, Frequency-Division Multiplexing (FDM), Orthogonal FDM (OFDM), Time-Division Multiplexing (TDM), Time-Division Multiple Access (TDMA), Extended TDMA (E-TDMA), General Packet Radio Service (GPRS), extended GPRS, Code-Division Multiple Access (CDMA), Wideband CDMA (WCDMA), CDMA 2000, Single-Carrier CDMA, Multi-Carrier CDMA, Multi-Carrier Modulation (MDM), Discrete Multi-Tone (DMT), BluetoothTM, Global Positioning System (GPS), Wi-Fi, Wi-Max, ZigBeeTM, Ultra-Wideband (UWB), Global System for Mobile Communication (GSM), 2G, 2.5G, 3G, 3.It may include a wireless connection based on one or more wireless communication protocols such as 5G, 4G, 5th generation (5G) mobile network, 3GPP, Long Term Evolution (LTE), LTE advanced, EDGE (Enhanced Data rates for GSM Evolution), etc. For example, the electronic device (500) may determine whether there is a Bluetooth connection with the electronic device of the real object. If the electronic device (500) is not connected to the electronic device of the real object, the electronic device (500) may perform operation 850. If the electronic device (500) is connected to the electronic device of the real object, the electronic device (500) may perform operation 840.

[0164] In operation 840, the electronic device (500) may generate a virtual object by mirroring the electronic device. For example, if the electronic device (e.g., a laptop) of a real object and the electronic device (500) are connected via Bluetooth, the electronic device (500) may generate a virtual object by mirroring the screen of the laptop without performing image processing on the screen of the laptop (e.g., image processing according to operations 705 to 720 of FIG. 7). That is, operations 820 to 840 may be for generating a virtual object through mirroring, not image processing. Hereinafter, operations for determining the location of the generated virtual object after the virtual object is generated through image processing and / or mirroring will be described.

[0165] In operation 850, the electronic device (500) may determine whether the real object is located within the user action radius. The real object may be an object corresponding to the virtual object generated through operations 810 to 840. The user action radius corresponds to a specific physical range from the user's location in the real space, and specifically, may include a range that is within a certain range based on the direction in which the user's upper body is facing and that is physically within the user's hand's reach. However, the user action radius may be set differently from the example above. If the real object is included in the user action radius in the real space (e.g., a specific physical range from the user's location in the real space), the electronic device (500) may perform operation 860. If the real object is not included in the user action radius in the real space (e.g., a specific physical range from the user's location in the real space), the electronic device (500) may perform operation 870.

[0166] In operation 860, the electronic device (500) may determine a virtual object (e.g., corresponding to the real object) as a world-lock object if the real object is within a specific physical range from the user's location in the real space. If the real object is within a specific physical range from the user's location in the real space, the electronic device (500) may determine the location of the virtual object as a location corresponding to the real object (e.g., overlaying on the real object or a location close to the real object) by determining the user's intention (e.g., wanting to duplicate the object and place it in a predetermined location). The world-lock object may be an object fixed in a global coordinate system centered on a preset location in the virtual environment. The preset location may be set to the user's location when the electronic device (500) boots up, but is not limited thereto. The location of the virtual object may be fixed in the virtual space as the virtual object is determined to be a world-lock object. For example, the virtual object may be fixed to a first space in the virtual space. In this case, if the user views the first space through the electronic device (500), the virtual object may be displayed in an image of the virtual space (e.g., output through the display of the electronic device (500)). However, if the user does not view the first space through the electronic device (500), the virtual object may not be displayed in an image of the virtual space.

[0167] In operation 870, the electronic device (500) may determine a virtual object as a body-lock object if the real object is not within a specific physical range from the user's position in the real space. If the real object is not within a specific physical range from the user's position in the real space, the electronic device (500) may determine the position of the virtual object as a position within the user's FOV by determining the user's intention (e.g., wanting to duplicate the object and place it within the field of view). The electronic device (500) may track the user's gaze and determine the position of the virtual object (e.g., the position of the virtual object to be displayed in the second image) as a position corresponding to the user's gaze.

[0168] According to one embodiment, a body lock object may be an object fixed in a coordinate system centered on the real-time location of the user. The location of the virtual object may vary depending on the real-time location of the user, as the virtual object is determined as a body lock object. The virtual object is not fixed to a specific location in the virtual space, but may always be displayed in an image of the virtual space (e.g., output through the display of the electronic device (500)) even when the user moves. The body lock object has the highest priority among virtual objects and must be displayed to the user preferentially, covering all other real objects or virtual objects, so it must not obstruct the user's view. That is, the location of the body object may be determined to be a location off the center of the user's display. For example, when the user views a first space through the electronic device (500), the virtual object may be displayed in an image of the virtual space (e.g., output through the display of the electronic device (500)). Additionally, even when the user moves to a location and views a second space (e.g., a space different from the first space in the virtual space), the virtual object may be displayed in an image (e.g., output through the display of the electronic device (500)) regarding the virtual space.

[0169]

[0170] FIG. 9 is a diagram illustrating a method for determining the size and depth of a virtual object according to one embodiment.

[0171] Referring to FIG. 9, according to one embodiment, operations 910 to 950 may be performed sequentially, but are not necessarily performed sequentially. For example, the order of each operation (910 to 950) may be changed, and at least two operations may be performed in parallel.

[0172] In operation 910, an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable device (103) of FIG. 2A, and / or the electronic device (500) of FIG. 5) may determine a location at which to display a virtual object. This is substantially the same as operations 850 to 870 of FIG. 8, and thus, redundant descriptions will be omitted.

[0173] In operation 920, the electronic device (500) may determine the content type of the virtual object. The electronic device (500) may classify the content type of the virtual object through an image classification engine. For example, the electronic device (500) may determine the ratio of images and text included in the virtual object. The electronic device (500) may compare the ratio of images and text included in the virtual object. If the ratio of images is large, the electronic device (500) may determine the content type of the virtual object as an image-related type. If the ratio of text is large, the electronic device (500) may determine the content type of the virtual object as a text-related type. If the content type of the virtual object is text-related content, the electronic device (500) may perform operation 930. If the content type of the virtual object is image-related content, the electronic device (500) may perform operation 940.

[0174] In operation 930, the electronic device (500) may determine the size and depth of the virtual object based on the readability of the virtual object when the content type is related to text. The electronic device (500) may set the size and depth considering user readability. For example, considering user readability, the depth of the virtual object may be determined as the topmost depth. In addition, the size of the virtual object may be determined as an optimal size considering user readability (e.g., a size that the user can read best). In this case, the size that the user can read best may be analyzed and calculated through an AI module (e.g., a generation module (535) of FIG. 5) based on previously accumulated user gaze information (e.g., acquired through a camera of the electronic device (500) (e.g., a camera module (180) of FIG. 1 and / or a camera (425) of FIG. 4)) and accumulated interaction information with real objects and virtual objects.

[0175] In operation 940, if the content type is related to an image, the electronic device (500) may determine the size and depth of the virtual object to correspond to the size and depth of the virtual reality object. The electronic device (500) may determine the size and depth of the virtual object to correspond to the size and depth of the real object. For example, if the real object corresponding to the virtual object is a painting, the electronic device (500) may identify that the content type of the virtual object is an image. The electronic device (500) may determine the size and depth of the virtual object to be the same as the size and depth of the real object (e.g., the painting). However, the present invention is not limited thereto, and when determining the size of the virtual object, the electronic device (500) may consider not only the real object but also other virtual objects (e.g., virtual objects displayed together in a second image (e.g., an image regarding a virtual space)). For example, the electronic device (500) may determine the size and depth of the virtual object by considering the positions of the real object and the virtual object displayed in the second image. The electronic device (500) may also determine the size and depth of the virtual object based on the relative distance between the user's location and real objects and / or the relative size of other virtual objects displayed in the second image.

[0176] In operation 950, the electronic device (500) may adjust the position, size, and depth of the virtual object based on additional user input and / or past history information.

[0177] According to one embodiment, if there is past history information on when a virtual object was displayed, the electronic device (500) can preferentially determine the position, size, and depth of the virtual object based on the past history information. The electronic device (500) can set (or adjust) the position, size, and depth of the virtual object to be the same as the past history information. The electronic device (500) can store the history of the user's use of the virtual object and analyze the user pattern of the use of the virtual object. The electronic device (500) can analyze the history of virtual objects frequently used by the user. If there is a history of frequent use of a virtual object by the user according to a user input, the electronic device (500) can display the virtual object in the same way as the user previously used it, based on the history of the virtual object. In addition, the electronic device (500) can give the display of a virtual object frequently used by the user a high priority so that it is displayed with priority over a plurality of virtual objects (e.g., virtual objects displayed at a previous point in time) (e.g., displaying a virtual object with a high priority without obscuring it). The electronic device (500) can analyze user patterns based on the user's gaze information and / or gesture information. The electronic device (500) can analyze and store information on frequently used virtual objects displayed to the user based on the user patterns, thereby determining the priority of the virtual objects.

[0178] According to one embodiment, when there is additional user input, the electronic device (500) can adjust the position, size, and depth of the virtual object in response to the user input. When there is user input (e.g., a request to reduce the size of the virtual object), the electronic device (500) can adjust the size of the virtual object to a size smaller than the size determined in operation 930 and / or operation 940, based on the user input. A separate interface may be provided in the second image for additional user input.

[0179]

[0180] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the wearable device (103) of FIG. 2A, and / or the electronic device (500) of FIG. 5) may include at least one processor (e.g., the processor (120) of FIG. 1 and the processor (410) of FIG. 4) including a processing circuit. The electronic device (101, 103, 500) may include a memory (e.g., the memory (130) of FIG. 1 and the memory (415) of FIG. 4) that stores instructions. The instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to generate a second image that composites the real space and the virtual space based on a first image of the real space captured by the camera. The instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to generate a virtual object corresponding to a real object included in the first image based on a user's input. The instructions may include: The instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to determine a location at which the virtual object is to be displayed within the second image based on whether the real object is contained within a specific physical range from the user's location in the real space. The instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to determine a size and depth of the virtual object based on a content type of the virtual object.The above instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to display the virtual object on the second image according to the location at which the virtual object is to be displayed and the size and depth of the virtual object.

[0181] According to one embodiment, the instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to detect a real object included in the first image. The instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to identify a planar area of ​​the real object. The instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to output an interface proposing conversion of the planar area into the virtual object.

[0182] According to one embodiment, the instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to generate a crop image that crops the planar area when the user's input is obtained through the interface. The instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to convert the planar area into the virtual object based on the crop image.

[0183] According to one embodiment, the instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to up-scale the crop image to improve the image quality of the virtual object.

[0184] According to one embodiment, the instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to determine the virtual object as a body-lock object if the real object is not within a specific physical range from the user's location in the real space. The body-lock object may be an object fixed in a coordinate system centered on the user's real-time location.

[0185] According to one embodiment, the instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to determine the virtual object as a world-lock object if the real object is contained within a specific physical range from a location of the user in the real space. The world-lock object may be an object fixed in a global coordinate system centered on a preset location in the virtual environment.

[0186] According to one embodiment, the instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to determine a size and depth of the virtual object based on readability of the virtual object when the content type is text-related.

[0187] According to one embodiment, the instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to determine a size and depth of the virtual object to correspond to a size and depth of the real object, when the content type relates to an image.

[0188] According to one embodiment, the instructions may be individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to preferentially determine a location to display the virtual object, a size and a depth of the virtual object based on past history information, if there is past history information on which the virtual object was displayed.

[0189] According to one embodiment, a method of operating an electronic device (e.g., the electronic device 101 of FIG. 1, the wearable device 103 of FIG. 2A, and the electronic device 500 of FIG. 5) may include an operation of generating a second image that aligns a real space and a virtual space based on a first image of a real space captured by a camera. The method may include an operation of generating a virtual object corresponding to a real object included in the first image based on a user's input. The method may include an operation of determining a location at which the virtual object is to be displayed in the second image based on whether the real object is included within a specific physical range from a location of the user in the real space. The method may include an operation of determining a size and a depth of the virtual object based on a content type of the virtual object. The method may include an operation of displaying the virtual object in the second image based on the location at which the virtual object is to be displayed and the size and depth of the virtual object.

[0190] According to one embodiment, the operation of generating a virtual object corresponding to the real object may include an operation of detecting the real object included in the first image. The operation of generating a virtual object corresponding to the real object may include an operation of identifying a planar area of ​​the real object. The operation of generating a virtual object corresponding to the real object may include an operation of outputting an interface proposing conversion of the planar area into the virtual object.

[0191] According to one embodiment, the operation of generating a virtual object corresponding to the real object may include an operation of generating a cropped image of the planar area when the user's input is obtained through the interface. The operation of generating a virtual object corresponding to the real object may include an operation of converting the planar area into the virtual object based on the cropped image.

[0192] According to one embodiment, the operation of converting the planar area into the virtual object may include an operation of up-scaling the crop image to improve the image quality of the virtual object.

[0193] According to one embodiment, the operation of determining a location to display the virtual object within the second image may include an operation of determining the virtual object as a body-lock object if the real object is not within a specific physical range from the user's location in the real space. The body-lock object may be an object fixed to a coordinate system centered on the user's real-time location.

[0194] According to one embodiment, the operation of determining a location to display the virtual object within the second image may include an operation of determining the virtual object as a world-locked object if the real object is contained within a specific physical range from the user's location in the real space. The world-locked object may be an object fixed in a global coordinate system centered on a preset location in the virtual environment.

[0195] According to one embodiment, the operation of determining the size and depth of the virtual object may include an operation of determining the size and depth of the virtual object based on readability of the virtual object when the content type is related to text.

[0196] According to one embodiment, the operation of determining the size and depth of the virtual object may include an operation of determining the size and depth of the virtual object to correspond to the size and depth of the real object, when the content type relates to an image.

[0197] According to one embodiment, the operating method may include, if there is past history information on which the virtual object was displayed, an operation of preferentially determining a location to display the virtual object and a size depth of the virtual object based on the past history information.

[0198]

[0199] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0200] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0201] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0202] Various embodiments of the present document may be implemented as software (e.g., a program (120)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0203] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0204] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices (101, 103, 500), At least one processor (120, 410) comprising a processing circuit; and Memory for storing instructions (130, 415) Including, The above instructions are individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to: Based on a first image of a real space captured by a camera, a second image is generated by compositing the real space and the virtual space, Based on the user's input, a virtual object corresponding to a real object included in the first image is generated, Determine a location to display the virtual object in the second image based on whether the real object is included within a specific physical range from the user's location in the real space; Based on the content type of the virtual object, determine the size and depth of the virtual object, An electronic device (101, 103, 500) that displays the virtual object on the second image according to the location at which the virtual object is to be displayed, the size and depth of the virtual object.

2. In paragraph 1, The above instructions are individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to: Detecting a real object included in the first image, Identify the planar area of ​​the above real object, An electronic device (101, 103, 500) configured to output an interface proposing transformation of the above-mentioned flat area into the above-mentioned virtual object.

3. In either of paragraphs 1 and 2, The above instructions are individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to: When the user's input is obtained through the above interface, a cropped image is generated by cropping the flat area, An electronic device (101, 103, 500) that converts the flat area into the virtual object based on the cropped image.

4. In any one of paragraphs 1 to 3, The above instructions are individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to: An electronic device (101, 103, 500) that up-scales the cropped image to improve the image quality of the virtual object.

5. In any one of paragraphs 1 to 4, The above instructions are individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to: If the real object is not within a specific physical range from the user's location in the real space, the virtual object is determined to be a body-lock object; The above body lock object is, An electronic device (101, 103, 500) which is an object fixed in a coordinate system centered on the real-time location of the user.

6. In any one of paragraphs 1 to 5, The above instructions are individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to: If the real object is included within a specific physical range from the user's location in the real space, the virtual object is determined as a world-lock object, The above world lock object is, An electronic device (101, 103, 500) which is an object fixed in a global coordinate system centered around a preset location in a virtual environment.

7. In any one of paragraphs 1 to 6, The above instructions are individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to: An electronic device (101, 103, 500) that determines the size and depth of the virtual object based on the readability of the virtual object when the content type is related to text.

8. In any one of paragraphs 1 to 7, The above instructions are individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to: An electronic device (101, 103, 500) that determines the size and depth of the virtual object to correspond to the size and depth of the real object when the content type relates to an image.

9. In any one of paragraphs 1 to 8, The above instructions are individually or collectively executed by the at least one processor (120, 410) to cause the electronic device (101, 103, 500) to: An electronic device (101, 103, 500) that, if there is past history information in which the virtual object was displayed, preferentially determines the location to display the virtual object, the size and depth of the virtual object based on the past history information.

10. In the operating method of an electronic device (101, 103, 500), An operation of generating a second image that is a composite of the real space and the virtual space based on a first image of the real space captured by the camera; An action of generating a virtual object corresponding to a real object included in the first image based on a user's input; An operation of determining a location to display the virtual object in the second image based on whether the real object is included within a specific physical range from the user's location in the real space; An operation of determining the size and depth of the virtual object based on the content type of the virtual object; and An operation of displaying the virtual object on the second image according to the location at which the virtual object is to be displayed, the size and depth of the virtual object. A method of operation, comprising:

11. In paragraph 10, The action of creating a virtual object corresponding to the above real object is: An operation of detecting a real object included in the first image; An operation of identifying a planar area of ​​the above real object; and An action that outputs an interface proposing the transformation of the above-mentioned planar area into the above-mentioned virtual object. A method of operation, comprising:

12. In any one of paragraphs 10 and 11, The action of creating a virtual object corresponding to the above real object is: When the user's input is obtained through the interface, an operation of generating a cropped image by cropping the flat area; and An operation of converting the flat area into the virtual object based on the cropped image. A method of operation, further comprising:

13. In any one of paragraphs 10 to 12, The operation of converting the above-mentioned plane area into the above-mentioned virtual object is: An operation of upscaling the above cropped image to improve the image quality of the virtual object. A method of operation, comprising:

14. In any one of paragraphs 10 to 13, The operation of determining the location to display the virtual object in the second image is as follows: An action to determine the virtual object as a body-lock object when the real object is not within a specific physical range from the user's location in the real space. Including, The above body lock object is, A method of operation, wherein the object is fixed in a coordinate system centered on the real-time location of the user.

15. In any one of paragraphs 10 to 14, The operation of determining the location to display the virtual object in the second image is as follows: An operation of determining the virtual object as a world-lock object when the real object is included within a specific physical range from the user's location in the real space. Including, The above world lock object is, A method of motion in which an object is fixed in a global coordinate system centered on a preset location in a virtual environment.

Images