Electronic device, method, and computer readable medium for rendering an image

Abstract

In an embodiment, an electronic device is provided. The electronic device may include: at least one display; at least one processor including processing circuitry; and a memory storing instructions and including one or more storage media. When executed individually or jointly by the at least one processor, the instructions may instruct the electronic device to perform the following operations in response to the execution of an application configured to provide a two-dimensional image: obtaining depth information about a region in three-dimensional space to be rendered from the application; determining resolution information of the application based on the depth information about the region; converting image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual-image information corresponding to images for both eyes; and displaying the rendered image on the at least one display based on the dual-image information.

Description

Technical Field

[0001] This disclosure relates to electronic devices, methods, and computer-readable media for rendering images. Background Technology

[0002] To provide an enhanced user experience, an electronic device is being developed that offers augmented reality (AR) services, which display computer-generated information associated with external objects in the real world. The electronic device can provide augmented reality services to the user by using virtual objects that correspond to the user's context.

[0003] To aid in understanding the purposes of this disclosure, the above information may be provided as related technology. No argument or decision is made regarding whether any description above can be applied as prior art related to this disclosure. Summary of the Invention

[0004] Technical solution In an embodiment, an electronic device is provided. The electronic device may include at least one display; at least one processor including processing circuitry; and a memory including one or more storage media storing instructions. When executed individually or jointly by the at least one processor, the instructions cause the electronic device to perform the following operations: in response to the execution of an application configured to provide a two-dimensional image, obtaining depth information of a region in three-dimensional space to be rendered from the application; determining resolution information of the application based on the depth information of the region; converting image information corresponding to a two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes; and displaying the rendered image through the at least one display based on the dual image information.

[0005] In one embodiment, a method executed by an electronic device is provided. The method may include: in response to execution of an application configured to provide a two-dimensional image, obtaining depth information of a region in three-dimensional space to be rendered from the application; determining resolution information of the application based on the depth information of the region; converting image information generated based on the resolution information corresponding to the two-dimensional image of the application into dual image information corresponding to images for both eyes; and displaying the rendered image based on the dual image information.

[0006] In an embodiment, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include a memory configured to store instructions, the memory comprising one or more storage media. The instructions, when executed individually or jointly by the at least one processor, cause the electronic device to perform the following operations: in response to the execution of an application configured to provide a two-dimensional image, obtaining depth information of a region in three-dimensional space to be rendered from the application; determining resolution information of the application based on the depth information of the region; converting image information corresponding to a two-dimensional image of the application generated based on the resolution information into dual-image information corresponding to images for both eyes; and displaying the rendered image based on the dual-image information.

[0007] In one embodiment, an electronic device is provided. The electronic device may include: at least one display; and at least one processor, including processing circuitry. The at least one processor may be configured to: in response to execution of an application configured to provide a two-dimensional image, obtain depth information of a region in three-dimensional space to be rendered from the application; determine resolution information of the application based on the depth information of the region; convert image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes; and display the rendered image through the at least one display based on the dual image information.

[0008] In one embodiment, an electronic device is provided. The electronic device may include: a spatialization manager for obtaining spatial information of an application; a resolution manager for determining the resolution of the application; and a virtual space manager for providing an image of the application in virtual space to a display buffer. The spatialization manager may be configured to obtain depth information of a region in three-dimensional space to be rendered of the application. The resolution manager may be configured to determine the resolution information of the application based on the depth information of the region to be rendered, and provide the determined resolution information of the application to the application. The spatialization manager may be configured to convert image information generated based on the resolution information into dual-image information corresponding to images for both eyes via the application. The virtual space manager may be configured to provide a rendered image to the display buffer based on the dual-image information. Attached Figure Description

[0009] Figure 1 It is a block diagram of electronic devices in a network environment.

[0010] Figure 2a An example of a perspective view showing a wearable device.

[0011] Figure 2bExamples of one or more hardware components set in a wearable device are shown.

[0012] Figures 3a to 3b An example showing the appearance of a wearable device.

[0013] Figure 4 An example of a block diagram showing a wearable device.

[0014] Figure 5 An example block diagram of an electronic device used to display images in virtual space is shown.

[0015] Figure 6a and Figure 6b An example block diagram of an electronic device for controlling the resolution of an image in virtual space is shown.

[0016] Figure 7 This shows an example of an application displayed in virtual space.

[0017] Figure 8 An example of an image used for focus rendering is shown.

[0018] Figure 9 This shows an example of how the resolution changes based on the movement of an application in virtual space.

[0019] Figure 10 This illustrates the operational flow of an electronic device used to control the resolution of applications in virtual space. Detailed Implementation

[0020] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of another embodiment. Singular expressions may include plural expressions unless the context clearly indicates otherwise. The terms used herein (including technical or scientific terms) may have the same meaning as commonly understood by one of ordinary skill in the art described herein. Among the terminology used in this disclosure, terms defined in general dictionaries may be interpreted as having the same or similar meaning as in the context of related art, and are not to be construed as having an ideal or overly formal meaning unless explicitly defined herein. In some cases, even terms defined in this disclosure may not be construed as excluding embodiments of this disclosure.

[0021] In the various embodiments of this disclosure described below, hardware methods will be described as examples. However, since the various embodiments of this disclosure include techniques using both hardware and software, software-based methods are not excluded.

[0022] For ease of description, the following description exemplifies terms referring to space (e.g., plane, object, shape, surface, graphic, three-dimensional graphic, region, occupied area, location, depth, or distance), terms referring to distance (e.g., location, distance, depth, depth information, distance information, distance value, depth value, location information, location data, or depth data), terms referring to application (e.g., application, program, app activity, application activity, activity, function), terms referring to value (e.g., threshold, reference value, reference area, reference range, level, threshold, range, value, or area), terms used for calculating state (e.g., step, operation, or process), terms referring to network entities, terms referring to components of a device, etc. Therefore, this disclosure is not limited to the terms described below, and other terms with equivalent technical meanings may be used.

[0023] Furthermore, in this disclosure, the terms "greater than" or "less than" are used to determine whether a particular condition is met or achieved, but this is merely a description of examples and does not exclude descriptions of "greater than or equal to" or "less than or equal to". A condition described as "greater than or equal to" may be replaced by "greater than", a condition described as "less than or equal to" may be replaced by "less than", and a condition described as "greater than or equal to and less than" may be replaced by "greater than and less than or equal to". Additionally, in the following, "A" to "B" refers to at least one element from A (inclusive) to B (inclusive). In the following, "C" and / or "D" indicates that at least one of "C" or "D" is included, i.e., {"C", "D", and "C" and "D"}.

[0024] Figure 1 This is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments.

[0025] Reference Figure 1In network environment 100, electronic device 101 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or with at least one of electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, memory 130, input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, user identification module (SIM) 196, or antenna module 197. In some embodiments, at least one of the above components (e.g., connection terminal 178) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components described above (e.g., sensor module 176, camera module 180, or antenna module 197) may be implemented as a single integrated component (e.g., display module 160).

[0026] Processor 120 may run software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 connected to processor 120, and may perform various data processing or calculations. According to embodiments, as at least part of the data processing or calculations, processor 120 may store commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store the result data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., central processing unit (CPU) or application processor (AP)) or an auxiliary processor 123 (e.g., graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor central processor, or communication processor (CP)) that is operationally independent of or combined with the main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or may be implemented as part of the main processor 121.

[0027] When the main processor 121 is inactive (e.g., in sleep) state, the auxiliary processor 123 (rather than the main processor 121) can control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190), or when the main processor 121 is active (e.g., running an application), the auxiliary processor 123 can work with the main processor 121 to control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190). According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) functionally associated with the auxiliary processor 123. According to embodiments, the auxiliary processor 123 (e.g., a neural processing unit) may include hardware architecture dedicated to artificial intelligence model processing. Artificial intelligence models can be generated through machine learning. For example, such learning can be performed via electronic device 101 where artificial intelligence is performed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple layers of artificial neural networks. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include software structures in addition to hardware structures.

[0028] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, software (e.g., program 140) and input or output data for commands associated with it. Memory 130 may include volatile memory 132 or non-volatile memory 134.

[0029] The program 140 may be stored as software in the memory 130, and the program 140 may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0030] Input module 150 can receive commands or data from outside electronic device 101 (e.g., a user) that will be used by other components of electronic device 101 (e.g., processor 120). Input module 150 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).

[0031] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. According to an embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0032] Display module 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display module 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to an embodiment, display module 160 may include a touch sensor adapted to detect touch or a pressure sensor adapted to measure the intensity of the force caused by touch.

[0033] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or headphones of an external electronic device (e.g., electronic device 102) that is directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0034] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0035] Interface 177 may support one or more specific protocols used to enable electronic device 101 to connect directly (e.g., wired) or wirelessly to external electronic devices (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0036] Connection 178 may include a connector, through which electronic device 101 may be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, connection 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] The haptic module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) or electrical stimuli that can be recognized by a user through his touch or kinesthesia. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0038] Camera module 180 can capture still or moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0039] The power management module 188 manages the power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0040] Battery 189 can power at least one component of electronic device 101. According to an embodiment, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable rechargeable battery, or a fuel cell.

[0041] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more communication processors capable of operating independently of processor 120 (e.g., application processor (AP)) and support direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). One of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, 5G network, next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components separate from each other (e.g., multiple chips). The wireless communication module 192 can identify and verify the electronic device 101 in the communication network (such as the first network 198 or the second network 199) using user information (e.g., the International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0042] Wireless communication module 192 can support 5G networks following 4G networks and next-generation communication technologies (such as new radio (NR) access technologies). NR access technologies can support enhanced mobile broadband (eMBB), massive machine-type communication (mMTC), or ultra-reliable low-latency communication (URLLC). Wireless communication module 192 can support high-frequency bands (e.g., millimeter-wave bands) to achieve, for example, high data transmission rates. Wireless communication module 192 can support various technologies used to ensure performance in high-frequency bands, such as, for example, beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. Wireless communication module 192 can support various requirements specified in electronic device 101, external electronic devices (e.g., electronic device 104), or network systems (e.g., second network 199). According to an embodiment, the wireless communication module 192 may support peak data rates (e.g., 20 Gbps or greater) for implementing eMBB, lost coverage (e.g., 164 dB or less) for implementing mMTC, or U-plane latency (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip) for implementing URLLC.

[0043] Antenna module 197 can transmit or receive signals or power to or from the exterior of electronic device 101 (e.g., external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiating element formed of a conductive material or conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, antenna module 197 may include multiple antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas by, for example, communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, additional components besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally incorporated into antenna module 197.

[0044] According to various embodiments, antenna module 197 may form a millimeter-wave antenna module. According to embodiments, the millimeter-wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and multiple antennas (e.g., an array antenna), wherein the RFIC is disposed on or adjacent to a first surface (e.g., a bottom surface) of the printed circuit board and is capable of supporting a specified high-frequency band (e.g., a millimeter-wave band), and the multiple antennas are disposed on or adjacent to a second surface (e.g., a top or side surface) of the printed circuit board and are capable of transmitting or receiving signals in the specified high-frequency band.

[0045] At least some of the aforementioned components can be interconnected and communicate signals (e.g., commands or data) between them via inter-peripheral communication schemes (e.g., bus, general purpose input / output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).

[0046] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 connected to a second network 199. Each of electronic device 102 or electronic device 104 can be a device of the same type as electronic device 101, or a device of a different type. According to an embodiment, all or some operations that would be performed on electronic device 101 can be performed on one or more of external electronic devices 102, external electronic devices 104, or server 108. For example, if electronic device 101 is required to automatically perform a function or service, or is required to perform a function or service in response to a request from a user or another device, electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service, instead of running the function or service, or electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service in addition to running the function or service. Upon receiving the request, one or more external electronic devices may perform at least a portion of the requested function or service, or perform additional functions or services related to the request, and transmit the result of the execution to electronic device 101. Electronic device 101 may provide the result as at least a partial response to the request, with or without further processing of the result. For this purpose, technologies such as cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing may be used. Electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and / or neural networks. According to embodiments, external electronic device 104 or server 108 may be included in a second network 199. Electronic device 101 may be applied to intelligent services based on 5G communication technology or IoT-related technologies (e.g., smart homes, smart cities, smart cars, or healthcare).

[0047] In embodiments of this disclosure, electronic devices for displaying images in virtual space (e.g., Figure 1 The electronic device 101 may be a wearable device. Wearable device 101 may include a head-mounted display (HMD) worn on a user's head. Wearable device 101 may be referred to as a head-mounted device (HMD), head-mounted electronic device, glasses-type electronic device, video see-through (or visible see-through) (VST) device, extended reality (XR) device, virtual reality (VR) device, and / or augmented reality (AR) device. Although the appearance of wearable device 101 in the form of glasses is shown, the embodiments are not limited thereto. Reference will be made to... Figure 4An example of the hardware configuration included in wearable device 101 is described exemplarily. (Refer to...) Figure 2a , Figure 2b , Figure 3a and / or Figure 3b This describes a structural example of a wearable device 101 that can be worn on the head of user 110. The wearable device 101 may be referred to as an electronic device. For example, the electronic device may be coupled with an accessory (e.g., a strap) to attach to the user's head to form an HMD.

[0048] The wearable device 101 according to an embodiment can perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when user 110 wears the wearable device 101, the wearable device 101 may include at least one lens disposed adjacent to the user 110's eyes. The wearable device 101 can combine light emitted from the display of the wearable device 101 with ambient light passing through the lens. The display area of ​​the display may be formed in the lens through which the ambient light passes. Because the wearable device 101 combines the light emitted from the display with the ambient light, user 110 can view an image that blends real objects identified by ambient light and virtual objects formed by light emitted from the display. The aforementioned augmented reality, mixed reality, and / or virtual reality may be referred to as extended reality (XR).

[0049] The wearable device 101 according to an embodiment can perform functions related to video perspective (or visible perspective) (VST) and / or virtual reality (VR). For example, when user 110 wears the wearable device 101, the wearable device 101 may include a housing covering the user 110's eyes. In this state, the wearable device 101 may include a display disposed on a first surface of the housing facing the eyes. The wearable device 101 may include a camera disposed on a second surface opposite the first surface. By using the camera, the wearable device 101 can acquire images and / or videos representing ambient light. The wearable device 101 can output images and / or videos in the display disposed on the first surface to allow user 110 to recognize ambient light through the display. The display area (or display zone) (or effective area or effective region) of the display disposed on the first surface may be formed by one or more pixels included in the display. The wearable device 101 can composite virtual objects with images and / or videos output through the display to allow user 110 to recognize virtual objects together with real objects recognized by ambient light.

[0050] The wearable device 101 according to an embodiment can identify or recognize its location (or orientation) and / or orientation (or orientation) based on images (and / or videos) obtained (or acquired) using a camera. The wearable device 101 can obtain information about external space using one or more cameras and / or one or more sensors. This information may include geographic locations (e.g., Global Positioning System (GPS) coordinates) of the external space identified from one or more sensors. This information may include images and / or videos of the external space identified from one or more cameras. The wearable device 101 can identify external objects included in the external space from the images and / or videos by performing object recognition on the images and / or videos.

[0051] In the following text, reference will be made to Figure 2a , Figure 2b , Figure 3a , Figure 3b and Figure 4 An example describing the hardware configuration of wearable device 101.

[0052] Figure 2a An example of a perspective view showing a wearable device. Figure 2b Examples of one or more hardware components disposed in a wearable device are shown. According to an embodiment, wearable device 101 may take the form of glasses that can be worn on a part of a user's body (e.g., the head). Figures 2a to 2b The wearable device 101 can be Figure 1 Examples of wearable devices 101. Wearable device 101 may include a head-mounted display (HMD). For example, the housing of wearable device 101 may include a flexible material (such as rubber and / or silicone) having a form that is tightly attached to a portion of the user's head (e.g., the portion around the two eyes on the face). For example, the housing of wearable device 101 may include one or more straps that can be wrapped around the user's head, and / or one or more temples that can be attached to the ears of the head.

[0053] Reference Figure 2a According to an embodiment, the wearable device 101 may include at least one display 250 and a frame 200 supporting the at least one display 250.

[0054] According to an embodiment, the wearable device 101 can be worn on a part of a user's body. The wearable device 101 can provide the user wearing the wearable device 101 with augmented reality (AR), virtual reality (VR), or mixed reality (MR), a combination of augmented reality and virtual reality. For example, the wearable device 101 can respond to... Figure 2b The motion recognition cameras 260-2 and 260-3 acquire the user's preset gestures, which are then displayed on at least one display 250. Figure 2bVirtual reality images provided by at least one optical device 282 and 284.

[0055] According to an embodiment, at least one display 250 may provide visual information to a user. For example, at least one display 250 may include a transparent lens or a semi-transparent lens. At least one display 250 may include a first display 250-1 and / or a second display 250-2 spaced apart from the first display 250-1. For example, the first display 250-1 and the second display 250-2 may be positioned at locations corresponding to the user's left and right eyes, respectively.

[0056] Reference Figure 2b At least one display 250 can provide a user with visual information transmitted from ambient light via lenses included in at least one display 250, as well as other visual information different from the visual information. The lenses can be formed based on at least one of Fresnel lenses, pancake lenses, or multi-channel lenses. For example, at least one display 250 may include a first surface 231 and a second surface 232 opposite to the first surface 231. A display area may be formed on the second surface 232 of at least one display 250. When a user wears the wearable device 101, ambient light can be transmitted to the user by incident on the first surface 231 and passing through the second surface 232. As another example, at least one display 250 can display, on a display area formed on the second surface 232, an augmented reality image combining a virtual reality image provided by at least one optical device 282 and 284 with a reality screen transmitted via ambient light.

[0057] According to an embodiment, at least one display 250 may include at least one waveguide 233 and 234 that transmits light emitted from at least one optical device 282 and 284 to a user via diffraction. The at least one waveguide 233 and 234 may be formed based on at least one of glass, plastic, or polymer. Nanopatterns may be formed on at least a portion of the exterior or interior of the at least one waveguide 233 and 234. The nanopatterns may be formed based on a grating structure having a polygonal or curved shape. Light incident on one end of the at least one waveguide 233 and 234 may be propagated through the nanopattern to the other end of the at least one waveguide 233 and 234. The at least one waveguide 233 and 234 may include at least one of at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and a reflective element (e.g., a mirror). For example, at least one waveguide 233 and 234 may be disposed in a wearable device 101 to guide the screen displayed by the at least one display 250 toward the user's eyes. For example, the screen can be sent to the user's eyes based on total internal reflection (TIR) ​​generated in at least one waveguide 233 and 234.

[0058] Wearable device 101 can analyze objects included in real images collected by cameras 260-4, combine them with virtual objects corresponding to objects that become subjects provided by augmented reality, and display them on at least one display 250. The virtual objects may include at least one of text and images for associating various information with the objects included in the real images. Wearable device 101 can analyze objects based on multiple cameras, such as stereo cameras. For object analysis, wearable device 101 may use multiple cameras and / or Time-of-Flight (ToF) to perform spatial recognition (e.g., Simultaneous Localization and Mapping (SLAM)). A user wearing wearable device 101 can view the images displayed on at least one display 250.

[0059] According to an embodiment, the frame 200 may be configured with a physical structure that allows the wearable device 101 to be worn on a user's body. According to an embodiment, the frame 200 may be configured such that when the user wears the wearable device 101, the first display 250-1 and the second display 250-2 are positioned corresponding to the user's left and right eyes. The frame 200 may support at least one display 250. For example, the frame 200 may support the first display 250-1 and the second display 250-2 at positions corresponding to the user's left and right eyes.

[0060] Reference Figure 2a According to an embodiment, the frame 200 may include an area 220 that at least partially contacts a portion of the user's body when the user wears the wearable device 101. For example, the area 220 of the frame 200 that contacts a portion of the user's body may include areas that contact a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that contact the wearable device 101. According to an embodiment, the frame 200 may include a nose pad 210 that contacts a portion of the user's body. When the wearable device 101 is worn by the user, the nose pad 210 may contact a portion of the user's nose. The frame 200 may include a first temple 204 and a second temple 205 that contact another portion of the user's body, different from a portion of the user's body.

[0061] For example, frame 200 may include a first frame 201 surrounding at least a portion of a first display 250-1, a second frame 202 surrounding at least a portion of a second display 250-2, a nose bridge 203 disposed between the first frame 201 and the second frame 202, a first pad 211 disposed along a portion of the edge of the first frame 201 from one end of the nose bridge 203, a second pad 212 disposed along a portion of the edge of the second frame 202 from the other end of the nose bridge 203, a first temple 204 extending from the first frame 201 and secured to a portion of the wearer's ear, and a second temple 205 extending from the second frame 202 and secured to a portion of the ear opposite the first ear. The first pad 211 and the second pad 212 may contact a portion of the user's nose, and the first temple 204 and the second temple 205 may contact a portion of the user's face and a portion of the user's ear. Temples 204 and 205 can be... Figure 2b Hinges 206 and 207 are rotatably connected to the frame. A first temple 204 is rotatably connected relative to the first frame 201 via a first hinge unit 206 disposed between the first frame 201 and the first temple 204. A second temple 205 is rotatably connected relative to the second frame 202 via a second hinge unit 207 disposed between the second frame 202 and the second temple 205. According to an embodiment, the wearable device 101 can identify external objects touching the frame 200 (e.g., a user's fingertips) and / or gestures performed by external objects by using touch sensors, grip sensors, and / or proximity sensors formed on at least a portion of the surface of the frame 200.

[0062] According to embodiments, wearable device 101 may include hardware that performs various functions (e.g., based on...). Figure 4 The block diagram describes the hardware. For example, the hardware may include a battery module 270, an antenna module 275, at least one optical device 282 and 284, a speaker (e.g., speaker 255-1 and 255-2), a microphone (e.g., microphone 265-1, 265-2 and 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) 290 (e.g., a printed circuit board). Various hardware components may be disposed within the frame 200.

[0063] According to an embodiment, the microphones of the wearable device 101 (e.g., microphones 265-1, 265-2, and 265-3) can acquire sound signals by being disposed on at least a portion of the frame 200. Figure 2b The image shows a first microphone 265-1 mounted on the nose bridge 203, a second microphone 265-2 mounted on the second frame 202, and a third microphone 265-3 mounted on the first frame 201, but the number and arrangement of the microphones 265 are not limited to... Figure 2b In embodiments where the number of microphones 265 included in the wearable device 101 is two or more, the wearable device 101 can identify the direction of the sound signal by using multiple microphones disposed on different parts of the frame 200.

[0064] According to an embodiment, at least one optical device 282 and 284 can project virtual objects onto at least one display 250 to provide various image information to a user. For example, at least one optical device 282 and 284 can be a projector. At least one optical device 282 and 284 can be disposed adjacent to at least one display 250, or can be included as part of at least one display 250. According to an embodiment, wearable device 101 may include a first optical device 282 corresponding to a first display 250-1 and a second optical device 284 corresponding to a second display 250-2. For example, at least one optical device 282 and 284 may include a first optical device 282 disposed on the periphery of the first display 250-1 and a second optical device 284 disposed on the periphery of the second display 250-2. The first optical device 282 can transmit light to a first waveguide 233 disposed on the first display 250-1, and the second optical device 284 can transmit light to a second waveguide 234 disposed on the second display 250-2.

[0065] In an embodiment, camera 260 may include a capturing camera 260-4, an eye-tracking camera (ETCAM) 260-1, and / or motion recognition cameras 260-2 and 260-3. The capturing camera 260-4, eye-tracking camera 260-1, and motion recognition cameras 260-2 and 260-3 may be positioned at different locations on frame 200 and perform different functions. Eye-tracking camera 260-1 may output data indicating the location of the eyes or gaze of a user wearing wearable device 101. For example, wearable device 101 may detect a gaze from an image including the user's pupils obtained through eye-tracking camera 260-1. Wearable device 101 may identify an object (e.g., a real object and / or a virtual object) focused by the user by using the user's gaze obtained through eye-tracking camera 260-1. Wearable device 101 that identifies the focused object may perform functions for interaction between the user and the focused object (e.g., gaze interaction). Wearable device 101 can represent the portion corresponding to the eyes of the user's avatar in a virtual space by using the user's gaze obtained through eye-tracking camera 260-1. Wearable device 101 can render an image (or screen) displayed on at least one display 250 based on the position of the user's eyes. For example, the visual quality (e.g., resolution, brightness, saturation, grayscale, and PPI) of a first region associated with the gaze within the image and the visual quality of a second region different from the first region may differ. Wearable device 101 can obtain an image with the visual quality of the first region and the second region matching the user's gaze by using focused rendering. For example, when wearable device 101 supports iris recognition, user authentication can be performed based on iris information obtained using eye-tracking camera 260-1. Figure 2b The illustration shows an example of an eye-tracking camera 260-1 facing the user's right eye, but the embodiment is not limited thereto, and the eye-tracking camera 260-1 may be facing the user's left eye alone or may be facing both eyes.

[0066] In an embodiment, camera 260-4 can capture a real image or background to be matched with a virtual image to enable augmented reality or mixed reality content. Camera 260-4 can be used to obtain images with high resolution based on high resolution (HR) or photo-video (PV). Camera 260-4 can capture images of specific objects present at the user's viewing location and can provide the images to at least one display 250. At least one display 250 can display an image overlaid with information about the virtual image provided by at least one optical device 282 and 284, and information about the real image or background including images of specific objects obtained using camera 260-4. Wearable device 101 can compensate for depth information (e.g., the distance between wearable device 101 and external objects obtained by a depth sensor) by using images obtained by camera 260-4. Wearable device 101 can perform object recognition by using images obtained by camera 260-4. Wearable device 101 can perform functions such as focusing on objects (or subjects) within an image (e.g., autofocus (AF)) and / or optical image stabilization (OIS) (e.g., image stabilization) using camera 260-4. When a screen representing virtual space is displayed on at least one display 250, wearable device 101 can perform a pass-through function to display an image obtained by camera 260-4 that overlaps with at least a portion of the screen. In an embodiment, camera 260-4 may be mounted on nose bridge 203, which is positioned between a first frame 201 and a second frame 202.

[0067] The eye-tracking camera 260-1 can achieve more realistic augmented reality by tracking the gaze of a user wearing the wearable device 101 and matching the user's gaze with visual information provided on at least one display 250. For example, when the user looks forward, the wearable device 101 can naturally display environmental information associated with the area in front of the user on at least one display 250 at the user's location. The eye-tracking camera 260-1 can be configured to capture images of the user's pupils to determine the user's gaze. For example, the eye-tracking camera 260-1 can receive gaze detection light reflected from the user's pupils and can track the user's gaze based on the position and movement of the received gaze detection light. In an embodiment, the eye-tracking camera 260-1 can be positioned corresponding to the user's left and right eyes. For example, the eye-tracking camera 260-1 can be positioned in a first frame 201 and / or a second frame 202 to face the direction in which the user wearing the wearable device 101 is located.

[0068] Motion recognition cameras 260-2 and 260-3 can provide specific events to a screen provided on at least one display 250 by recognizing the movement of the whole or part of a user's body (e.g., the user's torso, hands, or face). Motion recognition cameras 260-2 and 260-3 can obtain signals corresponding to the movement by recognizing the user's movement (e.g., gesture recognition) and can provide a display corresponding to that signal to at least one display 250. A processor can recognize signals corresponding to operation and can execute preset functions based on the recognition. Motion recognition cameras 260-2 and 260-3 can be used to perform SLAM of 6-DOF poses and / or spatial recognition functions using depth maps. A processor can perform gesture recognition functions and / or object tracking functions by using motion recognition cameras 260-2 and 260-3. In an embodiment, motion recognition cameras 260-2 and 260-3 can be mounted on a first frame 201 and / or a second frame 202.

[0069] The camera 260 included in the wearable device 101 is not limited to the eye-tracking camera 260-1 and motion recognition cameras 260-2 and 260-3 described above. For example, the wearable device 101 can identify external objects included in the field of view (FoV) by using a camera set towards the user. The identification of external objects by the wearable device 101 can be performed based on sensors (such as depth sensors and / or time-of-flight (ToF) sensors) used to identify the distance between the wearable device 101 and the external object. The camera 260 set towards the FoV may support autofocus (AF) and / or optical image stabilization (OIS) functions. For example, to obtain an image including the face of a user wearing the wearable device 101, the wearable device 101 may include a camera 260 set towards the face (e.g., a face-tracking (FT) camera).

[0070] Although not shown, the wearable device 101 according to an embodiment may also include a light source (e.g., an LED) that emits light toward an object captured by the camera 260 (e.g., the user's eyes, face, and / or an external object in FoV). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame 200 and hinge units 206 and 207.

[0071] According to an embodiment, the battery module 270 can power the electronic components of the wearable device 101. In an embodiment, the battery module 270 may be disposed in the first temple 204 and / or the second temple 205. For example, there may be multiple battery modules 270. The multiple battery modules 270 may be disposed on each of the first temple 204 and the second temple 205 respectively. In an embodiment, the battery module 270 may be disposed at the end of the first temple 204 and / or the second temple 205.

[0072] Antenna module 275 can transmit signals or power to the outside of wearable device 101, or can receive signals or power from the outside. In an embodiment, antenna module 275 may be disposed in the first temple 204 and / or the second temple 205. For example, antenna module 275 may be disposed close to a surface of the first temple 204 and / or the second temple 205.

[0073] The speaker 255 can output sound signals to the outside of the wearable device 101. The sound output module may be referred to as a speaker. In an embodiment, the speaker 255 may be disposed in the first temple 204 and / or the second temple 205 so as to be adjacent to the ear of the user wearing the wearable device 101. For example, the speaker 255 may include a second speaker 255-2 disposed adjacent to the user's left ear by being disposed in the first temple 204, and a first speaker 255-1 disposed adjacent to the user's right ear by being disposed in the second temple 205.

[0074] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state, or may emit light through operation corresponding to a specific state, in order to visually provide the user with information about a specific state of the wearable device 101. For example, when the wearable device 101 needs charging, it may emit red light at a constant period. In embodiments, the light-emitting module may be disposed on the first frame 201 and / or the second frame 202.

[0075] Reference Figure 2b According to an embodiment, the wearable device 101 may include a printed circuit board (PCB) 290. The PCB 290 may be included in at least one of a first temple 204 or a second temple 205. The PCB 290 may include an interposer layer disposed between at least two sub-PCBs. On the PCB 290, one or more hardware components (e.g., components manufactured by...) are included in the wearable device 101. Figure 4 The hardware shown in the different boxes can be configured. Wearable device 101 may include a flexible PCB (FPCB) for interconnecting the hardware.

[0076] According to an embodiment, the wearable device 101 may include at least one of a gyroscope sensor, a gravity sensor, and / or an accelerometer sensor for detecting the posture of the wearable device 101 and / or the posture of a body part (e.g., head) of the user wearing the wearable device 101. Each of the gravity sensor and the accelerometer sensor may measure gravitational acceleration and / or acceleration based on preset three-dimensional axes (e.g., x-axis, y-axis, and z-axis) perpendicular to each other. The gyroscope sensor may measure the angular velocity of each preset three-dimensional axis (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the accelerometer sensor, and the gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to an embodiment, the wearable device 101 may identify, based on the IMU, the user's movements and / or gestures performed to perform or stop a specific function of the wearable device 101.

[0077] Figures 3a to 3b An example showing the appearance of a wearable device (e.g., wearable device 101). Figures 3a to 3b The wearable device 101 can be Figure 1 An example of a wearable device 101. According to an embodiment, it can be... Figure 3a An example showing the appearance of the first surface 310 of the housing of the wearable device 101 is shown. Figure 3a And can Figure 3b An example showing the appearance of the second surface 320 opposite to the first surface 310 is shown. Figure 3b .

[0078] Reference Figure 3a According to an embodiment, the first surface 310 of the wearable device 101 may have an attachable shape on a user's body part (e.g., the user's face). Although not shown, the wearable device 101 may also include a strap for being secured to a user's body part and / or one or more temples (e.g., Figures 2a to 2b The first temple 204 and / or the second temple 205 of the wearable device 101 may be disposed on the first surface 310. A first display 250-1 for outputting an image to the left eye of the user and a second display 250-2 for outputting an image to the right eye of the user may be disposed on the first surface 310. The wearable device 101 may also include a rubber or silicone filler formed on the first surface 310 to prevent interference from light that is different from the light emitted from the first display 250-1 and the second display 250-2 (e.g., ambient light).

[0079] According to an embodiment, the wearable device 101 may include a camera 260-1 for capturing and / or tracking the two eyes of a user adjacent to each of the first display 250-1 and the second display 250-2. The camera 260-1 may be referred to as... Figure 2bThe wearable device 101 may include gaze-tracking camera 260-1. According to embodiments, the wearable device 101 may include cameras 260-5 and 260-6 for capturing and / or recognizing a user's face. Cameras 260-5 and 260-6 may be referred to as FT cameras. The wearable device 101 may control an avatar representing the user in virtual space based on the movement of the user's face recognized using cameras 260-5 and 260-6. For example, the wearable device 101 may alter the texture and / or shape of a portion of the avatar (e.g., a portion representing a human face) by using information obtained by cameras 260-5 and 260-6 (e.g., FT cameras) and representing the facial expressions of the user wearing the wearable device 101.

[0080] Reference Figure 3b Cameras (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, and 260-12) and / or sensors (e.g., depth sensor 330) for acquiring information associated with the external environment of the wearable device 101 may be positioned in conjunction with... Figure 3a The first surface 310 is positioned on the opposite second surface 320. For example, cameras 260-7, 260-8, 260-9, and 260-10 may be disposed on the second surface 320 to identify external objects. Cameras 260-7, 260-8, 260-9, and 260-10 may be referred to as... Figure 2b The motion recognition cameras 260-2 and 260-3.

[0081] For example, by using cameras 260-11 and 260-12, wearable device 101 can acquire images and / or videos to be sent to each of the user's two eyes. Camera 260-11 may be disposed on a second surface 320 of wearable device 101 to acquire an image to be displayed on a second display 250-2 corresponding to the right eye of the two eyes. Camera 260-12 may be disposed on the second surface 320 of wearable device 101 to acquire an image to be displayed on a first display 250-1 corresponding to the left eye of the two eyes. Cameras 260-11 and 260-12 may be referred to as... Figure 2b The camera used for shooting is 260-4.

[0082] According to an embodiment, the wearable device 101 may include a depth sensor 330 disposed on a second surface 320 to identify the distance between the wearable device 101 and an external object. By using the depth sensor 330, the wearable device 101 can obtain spatial information (e.g., a depth map) about at least a portion of the FoV of the user wearing the wearable device 101. Although not shown, a microphone for obtaining sound output from an external object may be disposed on the second surface 320 of the wearable device 101. According to an embodiment, the number of microphones may be one or more.

[0083] In the following text, and later referred to Figure 4 Describe the hardware or software configuration of wearable device 101.

[0084] Figure 4 An example of a block diagram showing a wearable device (e.g., wearable device 101). Figure 4 The wearable device 101 can be Figure 1 Electronic device 101 and Figures 2a to 3b Example of wearable device 101.

[0085] Reference Figure 4 The wearable device 101 according to the embodiment may include a processor 410, a memory 415, and a display 250 (e.g., Figure 2a , Figure 2b , Figure 3a and Figure 3b The first display 250-1 and / or the second display 250-2) and / or the sensor 420 (e.g., image sensor 421 and / or motion sensor 422). The processor 410, memory 415, display 250 and / or sensor 420 may be electrically connected and / or operatively connected to each other via electronic components such as communication bus 402. In this disclosure, the operative connection of the electronic components may include direct connections established between the electronic components and / or indirect connections established between the electronic components, such that a first electronic component in the electronic components is controlled by a second electronic component in the electronic components. The type and / or number of electronic components included in the wearable device 101 are not limited to, for example, Figure 4 As shown. For example, wearable device 101 may include only Figure 4 Some of the electronic components shown are shown.

[0086] According to embodiments, the processor 410 of the wearable device 101 may include circuitry (e.g., processing circuitry) for processing data based on one or more instructions. For example, the circuitry for processing data may include an arithmetic and logic unit (ALU), a field-programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In embodiments, the wearable device 101 may include one or more processors. The processor 410 may have a multi-core processor architecture (such as dual-core, quad-core, hexa-core, and / or octa-core). The multi-core processor architecture of the processor 410 may include an architecture based on multiple core circuits separated by power consumption, clock speed, and / or computational load per unit time (e.g., big-small architecture). In embodiments including a processor 410 with a multi-core processor architecture, the operations and / or functions of this disclosure may be performed individually or jointly by one or more cores included in the processor 410.

[0087] According to an embodiment, the memory 415 of the wearable device 101 may include electronic components for storing data and / or instructions input to and / or output from the processor 410. For example, the memory 415 may include volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM)). For example, the volatile memory may include at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). For example, the non-volatile memory may include at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, optical disk, and embedded multimedia card (eMMC). In embodiments, the memory 415 may be referred to as a storage device.

[0088] In an embodiment, the display 250 of the wearable device 101 can output visual information to the user of the wearable device 101. The display 250, positioned in front of the eyes of the user wearing the wearable device 101, can be disposed within at least a portion of the housing of the wearable device 101 (e.g., Figure 2a , Figure 2b , Figure 3a and Figure 3bThe first display 250-1 and / or the second display 250-2. For example, display 250 can output visual information to a user by being controlled by a processor 410 including circuitry such as a CPU 411, a graphics processing unit (GPU) 412, and / or a display processing unit (DPU) 413. Display 250 may include a flexible display, a flat panel display (FPD), and / or electronic paper. Display 250 may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light-emitting diodes (LEDs). LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, and for example, when wearable device 101 includes a lens for transmitting external light (or ambient light), display 250 may include a projector (or projection assembly) for projecting light onto the lens. In embodiments, display 250 may be referred to as a display panel and / or a display module. When worn by a user of wearable device 101, pixels included in display 250 may be oriented toward either of the user's two eyes. For example, display 250 may include a display area (or active area) corresponding to each of the user's two eyes.

[0089] In an embodiment, sensor 420 of wearable device 101 can generate electronic information that can be processed by processor 410 and / or memory 415 from non-electronic information associated with wearable device 101. For example, sensor 420 may include a Global Positioning System (GPS) sensor for detecting the geographic location of wearable device 101. In addition to GPS, sensor 420 may generate information indicating the geographic location of wearable device 101 based on Global Navigation Satellite Systems (GNSS) such as Galileo and BeiDou (compass) . The information may be stored in memory 415, processed by processor 410, and / or transmitted via communication circuitry to another electronic device different from wearable device 101.

[0090] Reference Figure 4As an example of a sensor 420 included in wearable device 101, an image sensor 421 and / or a motion sensor 422 are shown. Sensor 420 may include one or more optical sensors (e.g., charge-coupled device (CCD) sensors, complementary metal-oxide-semiconductor (CMOS) sensors) that generate electrical signals indicating the color and / or brightness of light. Image sensor 421 may be referred to as a camera. Multiple optical sensors included in image sensor 421 may be arranged in a two-dimensional array. Image sensor 421 may acquire electrical signals from each of the multiple optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to the light arriving at the two-dimensional array of optical sensors. For example, photographic data captured using image sensor 421 may indicate two-dimensional frame data acquired from image sensor 421. For example, video data captured using image sensor 421 may indicate a sequence of multiple two-dimensional frame data acquired from image sensor 421 according to the frame rate. Image sensor 421 may also include a flash located in the direction toward which image sensor 421 receives light, for outputting light in that direction.

[0091] According to an embodiment, the wearable device 101 may include a plurality of image sensors arranged in different directions, as exemplified by image sensor 421. (Refer to the above...) Figure 2a , Figure 2b , Figure 3a and Figure 3b The multiple image sensors may include Figure 2b Figure 3a An eye-tracking camera (e.g., configured to be positioned toward the eyes of a user wearing the wearable device 101) Figure 2b and Figure 3a An eye-tracking camera 260-1. Multiple image sensors may include outward-facing cameras. The processor 410 can identify the user's gaze direction by using images and / or video obtained from the eye-tracking camera. The eye-tracking camera may include an infrared (IR) sensor. The eye-tracking camera may be referred to as an eye sensor and / or an eye tracker.

[0092] For example, an outward-facing camera may be positioned facing the user wearing the wearable device 101 (e.g., the direction in which both eyes may be facing). The wearable device 101 may include multiple outward-facing cameras. Embodiments are not limited thereto, and the outward-facing cameras may be positioned facing external space. Using images and / or video obtained from the outward-facing cameras, the processor 410 may identify external objects. For example, the processor 410 may identify the position, shape, and / or posture (e.g., gesture) of the user's hands based on images and / or video obtained from the outward-facing cameras. Using images and / or video of the external environment obtained from the outward-facing cameras, the processor 410 may identify and / or track one or more objects in the external environment.

[0093] According to an embodiment, motion sensor 422 may output electrical signals indicating gravitational acceleration, acceleration, and / or angular velocity along multiple axes (e.g., x-axis, y-axis, and z-axis) perpendicular to each other and based on a designated origin within wearable device 101 and / or motion sensor 422. For example, processor 410 may repeatedly receive or acquire sensor data from motion sensor 422, including the magnitude of acceleration, angular velocity, and / or magnetic field along multiple axes, based on a specified period (e.g., 1 millisecond). In an embodiment, motion sensor 422 may be referred to as an inertial measurement unit (IMU). Sensors 420 included in wearable device 101 are not limited to those described above and may include grip sensors, proximity sensors, heart rate sensors, fingerprint sensors, illuminance sensors, and / or ToF sensors. Using motion sensor 422, processor 410 may detect motion of wearable device 101 (e.g., motion of wearable device 101 caused by a user wearing wearable device 101).

[0094] According to an embodiment, one or more instructions (or commands) indicating data to be processed by the processor 410 of the wearable device 101, and calculations and / or operations to be performed by the processor 410 of the wearable device 101, may be stored in the memory 415 of the wearable device 101. The set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, subroutine, and / or software application (hereinafter referred to as an application). For example, when a set of multiple instructions distributed in the form of an operating system, firmware, driver, program, and / or software application is executed, the wearable device 101 and / or processor 410 may perform... Figure 6a , Figure 6b , Figure 7 , Figure 8 , Figure 9 and Figure 10 At least one of the operations. In the following, a software application installed within wearable device 101 may mean that one or more instructions provided in the form of a software application (or package) are stored in memory 415, and that one or more applications are stored by processor 410 in an executable format (e.g., a file with an extension specified by the operating system of wearable device 101). As an example, an application may include programs and / or libraries associated with services provided to the user.

[0095] Reference Figure 4Depending on the objective, programs installed in wearable device 101 may be included in any of the different layers including application layer 440, frame layer 450, and / or hardware abstraction layer (HAL) 480. For example, programs (e.g., modules or drivers) designed to target the hardware of wearable device 101 (e.g., display 250 and / or sensor 420) may be included in hardware abstraction layer 480. Frame layer 450 may be referred to as XR frame layer in relation to including one or more programs for providing extended reality (XR) services. For example, Figure 4 The logically separated (or separated for ease of interpretation) layers shown may not represent that the address space of memory 415 is separated by layers.

[0096] For example, a program designed to target at least one of the hardware abstraction layer 480 and / or the application layer 440 (e.g., position tracker 471, spatial recognizer 472, gesture tracker 473, and gaze tracker 474) may be included in the frame layer 450. The program included in the frame layer 450 provides an application programming interface (API) capable of being executed (or invoked) based on another program.

[0097] For example, a program designed to target a user of wearable device 101 may be included in application layer 440. Extended Reality (XR) system user interface (UI) 441 and / or XR application 442 are shown as examples of programs included in application layer 440, but embodiments are not limited thereto. For example, a program included in application layer 440 (e.g., a software application) may perform functions supported by a program included in framework layer 450 by calling APIs.

[0098] For example, wearable device 101 may, based on the execution of XR system UI 441, display one or more visual objects on display 250 for performing interactions with the user. Visual objects may represent objects (such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables) that can be located within the screen for information transmission and / or interaction. Visual objects may be referred to as visual guides, virtual objects, visual elements, UI elements, view objects, and / or view elements. Wearable device 101 may, based on the execution of XR system UI 441, provide the user with functionalities available in a virtual space.

[0099] Reference Figure 4 The diagram shows that the XR system UI 441 includes, but is not limited to, a lightweight renderer 443 and / or an XR plugin 444. For example, the processor 410 may execute the lightweight renderer 443 and / or the XR plugin 444 in the frame layer 450 based on the XR system UI 441.

[0100] For example, wearable device 101 may, based on the execution of lightweight renderer 443, obtain resources (e.g., APIs, system processes, and / or libraries) for defining, generating, and / or executing a rendering pipeline that allows for partial modification. Lightweight renderer 443 may be referred to as a lightweight renderer pipeline in relation to defining a rendering pipeline that allows for partial modification. Lightweight renderer 443 may include renderers built prior to the execution of the software application (e.g., pre-built renderers). For example, wearable device 101 may, based on the execution of XR plugin 444, obtain resources (e.g., APIs, system processes, and / or libraries) for defining, generating, and / or executing the entire rendering pipeline. XR plugin 444 may be referred to as an open XR native client in relation to defining (or setting up) the entire rendering pipeline.

[0101] For example, wearable device 101 may display an image representing at least a portion of a virtual space on display 250 based on the execution of XR application 442. XR plugin 444-1 included in XR application 442 may include instructions supporting functions similar to those of XR plugin 444 in XR system UI 441. Descriptions overlapping with those of XR plugin 444 may be omitted in the description of XR plugin 444-1. Wearable device 101 may execute virtual space manager 451 based on the execution of XR application 442.

[0102] For example, wearable device 101 can display an image in virtual space on display 250 based on the execution of application 445. Application 445 can be configured to output image information for displaying two-dimensional images. Wearable device 101 can cause virtual space manager 451 to execute based on the execution of application 445. Wearable device 101 can generate dual image information for representing two-dimensional images in three-dimensional virtual space based on the execution of application 445. Here, taking into account the parallax of the two eyes, the dual image information may include a first image information for the left eye and a second image information for the right eye. In order to represent two-dimensional images in three-dimensional virtual space, wearable device 101 can generate dual image information based on the image information for displaying two-dimensional images.

[0103] According to an embodiment, wearable device 101 may provide virtual space services based on the execution of virtual space manager 451. For example, virtual space manager 451 may include a platform for supporting virtual space services. Wearable device 101 may, based on the execution of virtual space manager 451, identify a virtual space formed based on a user's location indicated by data obtained through sensor 430, and may display at least a portion of the virtual space on display 250. Virtual space manager 451 may be referred to as a compositional rendering manager (CPM).

[0104] For example, the virtual space manager 451 may include a runtime service 452. As an example, the runtime service 452 may be referred to as an OpenXR runtime module (or OpenXR runtime program). Based on the execution of the runtime service 452, the wearable device 101 may perform at least one of the following: user pose prediction, frame timing, and / or spatial input functions. As an example, based on the execution of the runtime service 452, the wearable device 101 may perform rendering of virtual space services for the user. For example, based on the execution of the runtime service 452, functions associated with a virtual space executable by the application layer 440 may be supported.

[0105] For example, the virtual space manager 451 may include a passthrough manager 453. The wearable device 101 may, based on the execution of the passthrough manager 453, display a screen representing the virtual space on the display 250 (e.g., Figure 1 When the screen (120) is used, images and / or videos representing the actual space obtained by an external camera are overlaid on at least a portion of the screen.

[0106] For example, the virtual space manager 451 may include an input manager 454. The wearable device 101 may, based on the execution of the input manager 454, identify data (e.g., sensor data) obtained by executing one or more procedures included in the perception service layer 470. The wearable device 101 may identify user input associated with the wearable device 101 by using the obtained data. User input may be associated with user movements (e.g., gestures), gaze, and / or voice identified by sensors 420 (e.g., image sensor 430 such as an external camera). User input may be identified based on external electronics connected (or paired) via communication circuitry.

[0107] For example, the Perception Abstraction Layer 460 can be used for data exchange between the Virtual Space Manager 451 and the Perception Service Layer 470. In relation to data exchange between the Virtual Space Manager 451 and the Perception Service Layer 470, the Perception Abstraction Layer 460 can be referred to as an interface. As an example, the Perception Abstraction Layer 460 can be referred to as OpenPX. The Perception Abstraction Layer 460 can be used for both perception clients and perception services.

[0108] According to an embodiment, the perception service layer 470 may include one or more programs for processing data obtained from the sensor 420. The one or more programs may include at least one of a position tracker 471, a spatial recognizer 472, a gesture tracker 473, and / or a gaze tracker 474. The type and / or number of the one or more programs included in the perception service layer 470 are not limited to, for example... Figure 4 As shown.

[0109] For example, wearable device 101 may identify its posture by using sensor 430 based on the execution of position tracker 471. Wearable device 101 may also identify its 6-DOF posture by using data obtained using external cameras (e.g., image sensor 421) and / or IMUs (e.g., motion sensor 422 including a gyroscope sensor, accelerometer, and / or geomagnetic sensor) based on the execution of position tracker 471. Position tracker 471 may be referred to as a head tracking (HeT) module (or head tracker or head tracking program).

[0110] For example, wearable device 101 may, based on the execution of spatial recognizer 472, obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of wearable device 101 (or the user of wearable device 101). Wearable device 101 may, based on the execution of spatial recognizer 472, reconstruct the surrounding environment of wearable device 101 in three dimensions using data obtained using an external camera (e.g., image sensor 421). Wearable device 101 may, based on the execution of spatial recognizer 472, identify at least one of planes, inclinations, and steps based on the three-dimensionally reconstructed surrounding environment of wearable device 101. Spatial recognizer 472 may be referred to as a scene understanding (SU) module (or scene recognition program).

[0111] For example, wearable device 101 may recognize (or identify) the hand gestures and / or hand postures of the user of wearable device 101 based on the execution of gesture tracker 473. For example, wearable device 101 may recognize the user's hand gestures and / or hand postures by using data obtained from an external camera (e.g., image sensor 421) based on the execution of gesture tracker 473. As an example, wearable device 101 may recognize the user's hand gestures and / or hand postures based on data (or images) obtained using an external camera based on the execution of gesture tracker 473. Gesture tracker 473 may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or gesture tracking module.

[0112] For example, wearable device 101 may identify (or track) the eye movements of its user based on the execution of gaze tracker 474. For example, wearable device 101 may identify the user's eye movements based on the execution of gaze tracker 474 by using data obtained from a gaze tracking camera (e.g., image sensor 421). Gaze tracker 474 may be referred to as an eye tracking (ET) module (or eye tracking program) and / or gaze tracking module.

[0113] Reference Figure 4CPU 411, graphics processing unit (GPU) 412, and / or display processing unit (DPU) 413 are shown as examples of processor 410. Renderer 490 may include instructions for rendering images in a three-dimensional virtual space. The processor 410 (e.g., DPU 413) executing renderer 490 may obtain at least one image to be at least partially displayed on a display area of ​​display 250 at a software application (e.g., a software application executed by CPU 411 and / or GPU 412). For example, processor 410 executing renderer 490 may determine the location of an area to be rendered for an application (e.g., XR application 242, application 245). Processor 410 executing renderer 490 may generate an image of the application to be displayed on display 250. Renderer 490 may generate a composite image to be displayed on display 250 by compositing the image.

[0114] For example, the processor 410 executing renderer 490 can divide the display area of ​​display 250 into a focused portion (or a focused area) and a peripheral portion (or a remaining area) using the gaze position calculated using position tracker 471 and / or gaze tracker 474. For example, processor 410, which detects the coordinate values ​​of the gaze position, can determine a portion of the display area including the coordinate values ​​as the focused area. The DPU 413 executing renderer 490 can obtain at least one image corresponding to each of the focused area and the remaining area, and has a size smaller than the size of the entire display area of ​​display 250 or a resolution smaller than the resolution of the display area.

[0115] The processor 410 of the renderer 490 can obtain or generate a composite image to be displayed on the display 250 by compositing an image corresponding to the focus area and an image corresponding to the peripheral portion. For example, the processor 410 can enlarge the image corresponding to the peripheral portion to the size of the entire display area of ​​the display 250 by performing magnification. The processor 410 can generate a composite image to be displayed on the display 250 by combining the image corresponding to the focus area with the magnified image. The processor 410 can blend the magnified image and the image corresponding to the focus area by applying visual effects (such as blurring) along the boundary lines of the image corresponding to the focus area.

[0116] Figure 5 An example block diagram of an electronic device (e.g., electronic device 101, wearable device 101) for displaying images in virtual space is shown. Figure 5The text describes an example of multiple programs / instructions being executed to display images in virtual space. These programs / instructions may all be executed on a single processor (e.g., an application processor, AP), or they may be executed by multiple processors (e.g., an AP, a graphics processing unit (GPU), a neural processing unit (NPU)). The meaning of "executed by multiple processors" indicates that some programs / instructions may be executed by a first processor, while other programs / instructions may be executed by a second processor, different from the first processor.

[0117] Reference Figure 5 The electronic device 101 can execute the virtual space manager 550 (e.g., Figure 4 The virtual space manager 451 (CPM) renders images in virtual space. For the virtual space manager 550, at least partially referencing... Figure 4 The description of the virtual space manager 451 is provided. The virtual space manager 550 may include a platform for supporting virtual space services. The virtual space manager 550 may include a runtime service 551 (e.g., an OpenXR runtime), a panel renderer 552 (e.g., a 2D panel renderer), and an XR compositor 553. The electronic device 101 may, based on the execution of the runtime service 551, perform at least one of the following user pose prediction functions, frame timing functions, and / or spatial input functions. For the runtime service 551, reference may be made at least in part to… Figure 4 The description of runtime service 452 is provided. Electronic device 101 may, based on the execution of panel renderer 552, display at least one image (video) on a panel (e.g., a 2D panel) to realize virtual space via a display. For example, electronic device 101 may display a rendered image corresponding to RGB information 566 from a panel of spatialization manager 540, which will be described later, via a display (e.g., display 250). Electronic device 101 may, based on the execution of XR compositor 553, composite an image of an actual area captured in virtual space by a camera (hereinafter, a pass-through image) and an image of the virtual area. For example, electronic device 101 may, based on the execution of XR compositor 553, generate a composite image by merging the pass-through image and the virtual area image. Electronic device 101 may send the generated composite image to a display buffer, causing the composite image to be displayed. Electronic device 101 may identify the virtual space via virtual space manager 550 and display at least a portion of the virtual space on display 250. Virtual space manager 550 may be referred to as CPM. The electronic device 101 can execute the virtual space manager 550 to render an image corresponding to at least a portion of the virtual space.

[0118] According to an embodiment, electronic device 101 may execute spatialization manager 540. Spatialization manager 540 may perform processing for displaying images in a three-dimensional virtual space. Electronic device 101 may perform preprocessing based on the execution of spatialization manager 540, enabling the rendering of images in a three-dimensional virtual space by virtual space manager 550. For example, electronic device 101 may perform... Figure 4 At least a portion of the functionality of renderer 490. Electronic device 101 may process image information provided by applications (e.g., XR application 510, application 520 providing a normal 2D screen other than XR, and application providing system UI 530) based on the execution of spatialization manager 540. Spatialization manager 540 (e.g., Space Flinger) may include system screen manager 541 (e.g., System scene), input manager 542 (e.g., Input Routing), and lightweight rendering engine 543 (e.g., Impress Engine). System screen manager 541 can be executed to display system UI 530. System UI related information 564 may be sent to system screen manager 541 from a program (e.g., API) providing system UI 530. System UI related information 564 may be obtained through spatializer API and / or in-process private API. Spatialization manager 540 can determine the layout (e.g., position, display order) of the system UI 530 screen in three-dimensional space using pre-allocated resources. System screen manager 541 can send image information 567 for rendering the system UI 530 screen to virtual space manager 550 according to the layout. Input manager 542 can be configured to process user input (e.g., user input on the system screen or application screen). Lightweight rendering engine 543 can be a renderer for generating images (e.g., lightweight renderer 443). For example, lightweight rendering engine 543 can be used to display system UI 530. According to an embodiment, spatialization manager 540 may include lightweight rendering engine 543 for rendering system UI. According to an embodiment, if lightweight rendering engine 543 does not have sufficient resources to render the avatar used in HMD, at least one external rendering engine can be used. In this case, to resolve compatibility issues with external rendering (e.g., third-party engines), an external rendering engine support module can be added within spatialization manager 540.

[0119] According to an embodiment, the electronic device can execute an application. For example, a virtual space manager 550 can be executed in response to the execution of an XR application 510 (e.g., XR application 442, 3D game, XR map, and other immersive applications). The electronic device 101 can provide dual-image information 561 provided from the XR application 510 to the virtual space manager 550. To display images in three-dimensional space, the dual-image information 561 can include two image information that takes into account the parallax of the two eyes. For example, the dual-image information 561 can include a first image information for the user's left eye and a second image information for the user's right eye for rendering in a three-dimensional virtual space. In the following, in this disclosure, dual-image information is used as a term for image information used to indicate images for both eyes in three-dimensional space. In addition to dual-image information, binocular image information, dual-image data, dual images, binocular image data, stereo image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimension conversion image data, binocular parallax image data, and / or equivalent technical terms may also be used. Electronic device 101 can generate a composite image by merging image layers via virtual space manager 550. Electronic device 101 can send the generated composite image to a display buffer. The composite image can be displayed on display 250 of electronic device 101.

[0120] According to an embodiment, the electronic device may execute at least one of an application 520 (e.g., a first application 520-1, a second application 520-2, ..., and an Nth application 520-N) that is different from the XR application 510. According to an embodiment, application 520 may be configured to output image information for displaying a two-dimensional image. In other words, application 520 may provide a two-dimensional image. As an example, application 520 may be an image application, a scheduling application, or an internet browser application. Assume that in response to the execution of application 520, image information 562 provided from application 210 is provided to the virtual space manager 550. Since image information 562 only has x and y coordinates in a two-dimensional plane, it may be difficult to consider the priority relationship of other applications relative to the user (i.e., the distance between them and the user). Even when displaying an application 520 that provides a general 2D screen, electronic device 101 may execute spatialization manager 540 to provide dual image information to virtual space manager 550. For example, electronic device 101 may receive application-related information 563 from first application 520-1 based on the execution of spatialization manager 540. For example, application-related information 563 may include image information (e.g., including RGB information per pixel) indicating a two-dimensional image of the first application 520-1 and / or content information in the first application 520-1 (e.g., characteristics of the content executed in the first application, the type of content). Application-related information 563 can be obtained via the spatializer API. Based on the execution of the spatializer 540, the electronic device 101 may identify the location of the region to be rendered in the first application 520-1 and information about the size of the region to be rendered (hereinafter, location information). Based on the execution of the spatializer 540, the electronic device 101 may generate dual image information 565 (e.g., RGBx2) that takes into account the parallax of the user's two eyes using the image information and location information. Based on the execution of the spatializer 540, the electronic device 101 may provide the dual image information 565 to the virtual spatializer 550. By converting a simple two-dimensional image into dual image information 565, the problem that arises when image information 562 is sent directly to the virtual spatializer 550 can be solved. Furthermore, since at least a portion of the function of displaying images in virtual space is performed by the spatialization manager 540 instead of the virtual space manager 550, the burden on the virtual space manager 550 is reduced. However, since image information from application 520 may not be transmitted directly to the virtual space manager 550, but rather through the spatialization manager 540, the quality of the final image output to the user may be reduced. As an example, an image may be rendered at a resolution of approximately 2756×1846 in the first application 520-1, but the image may be downsampled during transmission through the spatialization manager 540 to the virtual space manager 550 (e.g., downsampled from approximately 2756×1846 resolution to approximately 1160×680 resolution).Subsequently, the virtual spatial manager 550 can upsample the downsampled image (e.g., upsample from approximately 1160×680 resolution to approximately 1625×1070 resolution) and transmit the upsampled image to the display buffer. Thus, during the transmission of the image from application 520 to spatialization manager 540 and from spatialization manager 540 to virtual spatial manager 550, resolution mismatch, aliasing problems, or image quality degradation problems may occur during the upsampling process. To address these problems, this disclosure is based on... Figure 5 The system architecture shown describes the techniques used to control the resolution of the area to be displayed in the application and to perform focused rendering.

[0121] Figure 6a and Figure 6b An example block diagram of an electronic device (e.g., electronic device 101 or wearable device 101) for controlling the resolution of an image in virtual space is shown. Figure 6a and Figure 6b The text describes an example of executing multiple programs / instructions for controlling the resolution of an image in virtual space. These programs / instructions may all be executed on a single processor (e.g., an AP), or they may be executed by multiple processors (e.g., an AP, a graphics processing unit (GPU), and a neural processing unit (NPU)). Execution by multiple processors implies that a portion of the program / instructions may be executed by a first processor, while another portion may be executed by a second processor, different from the first processor.

[0122] Reference Figure 6aElectronic device 101 can execute spatialization manager 540 and virtual space manager 550 to render images in virtual space. For example, electronic device 101 can request rendering size, content type, and location information of the application to be executed from spatialization manager 540 in response to the execution of an application configured to output two-dimensional images (e.g., application 520) to generate dual image information 565 from the application's two-dimensional image (e.g., image information 563). In other words, electronic device 101 can be configured to convert image information 563 of an application other than an immersive application (e.g., XR application 510) (e.g., application 520) into dual image information 565. Dual image information 565 can be generated based on location information of the area to be rendered for the corresponding application and image information 563. Electronic device 101 can execute virtual space manager 550 to display dual image information in three-dimensional virtual space. In this disclosure, location information as the location of the application to be drawn in virtual space (i.e., the location to be rendered) can indicate xyz coordinates in three-dimensional space. Besides location information, location information can also be referred to as rendering region information, location region information, location information, spatial information, coordinate information, region information, depth information, 3D coordinate information, rendering information, distance information, z-information, and / or terms with equivalent technical meanings. For the Spatialization Manager 540 and Virtual Spatial Manager 550, please refer to... Figure 5 The description.

[0123] Electronic device 101 may execute resolution manager 630 to adaptively control resolution based on the location of the area rendered by application 520, the size of the area, and / or the characteristics of the content executed in application 520. In this disclosure, characteristics of content indicating type (such as whether the content executed in the application requires readability (e.g., a character requires higher readability than a photograph), or whether the user's level of interest is high (e.g., a face has a higher level of interest than an arm)) are hereinafter referred to as content type and are described. In addition to type, content type may be replaced by content information, characteristic information, object information, object type information, object characteristic information, sample information, image information, object category, object type, object characteristic, and / or equivalent technical terms. Electronic device 101 may determine the resolution of application 520 in response to the execution of resolution manager 630. For example, the electronic device 101 (e.g., spatialization manager 540) may request information about the application 520 to be executed at a given time (e.g., rendering size, content type, or location information), information about focus rendering (e.g., focus area, focus level, or focus method), and / or information about the user (e.g., the user's region of interest, gaze data, execution environment, or whether passthrough is being performed). The spatialization manager 540 may provide data 671, including the requested information, to the resolution manager 630 (e.g., a zoom or focus manager) based on the request.

[0124] According to an embodiment, resolution manager 630 may be configured to determine the resolution (e.g., final resolution (scaling factor) or focus level) for image rendering of application 520 based on data. According to an embodiment, resolution manager 630 may determine whether the type of content to be rendered in the application is a moving image and whether it includes characters requiring readability. As an example, if it includes characters requiring readability, electronic device 101 may set the resolution to a relatively high level. As an example, in the case of video or photographs, electronic device 101 may set the resolution to a relatively low level. Resolution manager 630 may provide resolution information 672 to application 520. Resolution information 672 may indicate rendering settings directed to application 520. Electronic device 101 may provide resolution information (e.g., resolution information 672) indicating the resolution to application 520. Resolution information may include scaling factors, resolution factors, resolution values, resolution levels, and / or equivalent technical parameters. Application 520 may change rendering settings based on the resolution information. Application 520 may change rendering settings based on the resolution information and provide an image based on the changed rendering settings. In the following, in this disclosure, scaling factor is used as an example of resolution information, but resolution factor, resolution adjustment parameter, resolution scaling factor, resolution control parameter, resolution parameter and / or equivalent technical terms may also be used instead of scaling factor.

[0125] According to an embodiment, electronic device 101 (e.g., resolution manager 630) can determine not only the resolution for image rendering of application 520, but also parameters for focus rendering. For example, electronic device 101 can determine the focus area and / or focus level based on the execution of resolution manager 630. The focus area can indicate the area to be displayed at high resolution in focus rendering (e.g., it may be referred to as the focus area, focus portion, region of interest, resolution focus area, etc.). The focus level can indicate the size of the focus area compared to the entire area. In the following, in this disclosure, in addition to focus level, focus level may be used with focus resolution level, resolution rendering level, focus rendering level, focus area ratio, focus area size, and / or equivalent technical terms. According to an embodiment, resolution manager 630 can determine the focus area (focus rendering) and / or focus level of the image to be ultimately displayed. According to an embodiment, resolution manager 630 can provide parameters (e.g., at least a portion of resolution information 672) to application 520. Electronic device 101 executing application 520 can determine rendering settings based on the provided information. The electronic device 101 can output the image layer to be finally rendered according to the rendering settings. The output image layer can be converted into dual image information by the spatialization manager 540. The converted dual image information can be rendered by the virtual spatial manager 550.

[0126] According to embodiments, the characteristics of the content displayed on an application can be changed when the application is executed and at the point in time when the application is executed. For example, when an application is simultaneously displayed to a user, an operation can be performed to place the application of interest in the position closest to the user. At this time, the efficiency of GPU resources can be improved by changing the resolution of the rendered image. As another example, an application (such as an internet browser) can display various types of content. As an example, the content to be rendered can be changed from a text page requiring readability to content that does not require readability (such as videos or photos). Even in this case, it may be necessary to adaptively adjust the resolution of the displayed image according to the type of content. According to embodiments, an electronic device (e.g., spatialization manager 540) can provide the changed content characteristics to a resolution manager 630. The resolution manager 630 can change the rendering settings of the application 520 being executed based on the received information. The electronic device 101 (e.g., resolution manager 630) can provide the rendering settings to the application 520. The spatialization manager 540 can convert the image information 563 of the application 520 into dual image information 565 according to the settings. The spatialization manager 540 can provide the dual image information 565 to the virtual spatial manager 550. As a non-limiting example, when all applications (e.g., XR application 510 and application 520) are executed and the focus level is adjusted, the dual image information 651 of XR application 520 can be provided to the virtual space manager 550 via the resolution manager 630.

[0127] Spatialization manager 540 may include a lightweight rendering engine 543 as a lightweight renderer 443 for rendering the system UI. However, while the lightweight rendering engine 543 may have sufficient resources to render the system UI 530, it may not have enough resources to render individual avatars. If an external rendering engine (e.g., avatar rendering engine 660) is used, there may be difficulties in displaying the avatar rendering results along with the system UI 530 because it is incompatible with the structure of spatialization manager 540.

[0128] To ensure compatibility with external rendering engines, the spatialization manager 540 may include a third-party support module 661. For example, an external rendering engine 660 may be necessary because various rendering requirements for the avatar to be rendered in the second application 520-2 (such as overall generation, deformation, clothing changes, and motion representation) may not be met. The electronic device 101 may collect rendering results from the external rendering engine (e.g., the avatar rendering engine 660) in response to the execution of the third-party support module 661 of the spatialization manager 540. Since the rendering results from the external rendering engine may be incompatible with the lightweight rendering engine 543, it may be necessary to convert the rendering results to a format output by the lightweight rendering engine 543 or to convert them into an integrated format for rendering. Through this process, the results of the external rendering engine can ultimately be represented by the lightweight rendering engine 543. For example, avatar data and system UI-related data output from the lightweight rendering engine 543 may be collected by the spatialization manager 540. Electronic device 101 can provide collected data to virtual space manager 550 in the format of dual image information 565, based on the execution of spatialization manager 540, for rendering with binocular parallax.

[0129] Reference Figure 6b The spatialization manager 540 can be configured to include the resolution manager 630. For example, the resolution manager 630 can be a component of the spatialization manager 540. According to an embodiment, Figure 6a The functions of the resolution manager 640 can be performed by the spatialization manager 540. For example, when all applications (e.g., XR application 510 and application 520) are executed, and the focus level is adjusted, the dual image information 565 of XR application 520 can be transferred to the virtual spatial manager 451 via the spatialization manager 540. Although not in Figure 6b As shown, however, if focus rendering is not performed (e.g., if focus level is not set), the dual image information 565 of the XR application 520 can be directly transmitted to the virtual space manager 451 without passing through the spatialization manager 540.

[0130] Figure 7 This shows an example of an application displayed in virtual space.

[0131] Reference Figure 7 According to an embodiment, electronic device 101 can execute applications. For example, electronic device 101 can execute a first application 710. The first application 710 may be an internet browser. Electronic device 101 can execute a second application 720. The second application 720 may be a calendar application.

[0132] According to an embodiment, electronic device 101 can display executed applications in a virtual space. Electronic device 101 can display rendered images corresponding to image information provided from each application via a display. According to an embodiment, electronic device 101 can determine the resolution of the images provided from the applications based on the size and / or location of the area in the virtual space where the rendered images are to be displayed (hereinafter referred to as the rendering area). The resolution may not indicate the resolution of the output image converted by image processing (e.g., downsampling or upsampling) in spatialization manager 540 or virtual space manager 550, but may represent the resolution of the image provided at the application level. For example, electronic device 101 can determine the resolution of the image of a first application 710. Electronic device 101 can determine the resolution based on a first depth 721, which is the distance from a reference point of electronic device 101 (e.g., the user's position) to the first area. Electronic device 101 can convert the image information (e.g., a two-dimensional image) of the first application 710 corresponding to the determined resolution into dual-image information (e.g., two two-dimensional images) for both eyes. Electronic device 101 can display the rendered images in the virtual space via a display (e.g., display 250) based on the dual-image information. Electronic device 101 can determine the resolution of the image of the second application 720. Electronic device 101 can determine the resolution based on a second depth 722, which serves as the distance from a reference point of electronic device 101 (e.g., the user's position) to the second region. Electronic device 101 can convert the image information (e.g., a two-dimensional image) of the second application 720 corresponding to the determined resolution into dual-image information (e.g., two two-dimensional images) for both eyes. Electronic device 101 can display the rendered image in virtual space based on the dual-image information via a display (e.g., display 250). In the three-dimensional virtual space, the second region rendering the second application 720 can be located relatively farther than the first region rendering the first application 710. Since the second depth 722 is deeper than the first depth 721, electronic device 101 can set the resolution of the first application 710 to be higher than the resolution of the second application 720. This is because it is assumed that the closer the activity is to the user, the higher the user's level of interest.

[0133] According to an embodiment, the application's resolution can be determined not only based on the location of the rendering area but also based on the type of content provided by the application. The resolution may not indicate the resolution of the output image converted by image processing (e.g., downsampling or upsampling) in the spatialization manager 540 or virtual spatial manager 550, but rather the resolution of the image provided at the application level. For example, if there are many characters to be read in the application's content, the electronic device 101 may set the application's resolution to be relatively high. As an example, the electronic device 101 may set the resolution of an e-book application to be higher than that of a video application. The electronic device 101 (e.g., spatialization manager 540) may obtain information about the content type from the application (e.g., application 520) and determine a scaling factor based on the application's resolution via the resolution manager 630. The electronic device 101 may request an image from the application again based on the determined scaling factor. The electronic device 101 may obtain an image (e.g., a two-dimensional image) generated according to the scaling factor from the application and convert the generated image into dual-image information having images for both eyes in three-dimensional space. The electronic device 101 may display the rendered image based on the dual-image information.

[0134] exist Figure 7 The present disclosure describes an example of determining the resolution of an image provided by an application based on the application's location or the characteristics of the application's content, but embodiments of this disclosure are not limited thereto. In addition to the information described above, electronic device 101 may also determine the resolution of an image provided by an application based on the user's gaze information. For example, electronic device 101 may use at least one sensor (e.g., Figure 2b and Figure 3a gaze tracking camera 260-1 or Figure 4 Image sensor 421 acquires gaze data. Electronic device 101 can set the resolution differently based on the distance between the location indicated by the gaze data and the location of the rendering area for each application. For example, when the direction to the rendering area is further away than the direction of the gaze data, electronic device 101 can set the resolution of the corresponding application to be lower. As another example, if the location indicated by the user's gaze data is within a threshold range from the location of the application's rendering area, electronic device 101 can determine the resolution of the image provided by the application based on the depth of the rendering area. If the location indicated by the user's gaze data is outside the threshold range from the location of the application's rendering area, electronic device 101 can determine the resolution of the image provided by the application based on the depth of the rendering area, according to a value set to a predetermined value. This is because, since it is outside the user's region of interest, not adjusting the resolution may be efficient in terms of resource management.

[0135] Figure 8An example image used for focus rendering is shown. Focus rendering can refer to a technique that divides an image into a focus region and a remaining region, composites a high-resolution image corresponding to the focus region and a low-resolution image corresponding to the remaining region, and renders the composite image.

[0136] Reference Figure 8 The electronic device 101 can execute an application (e.g., application 520). The application can be an application configured to provide a two-dimensional image. For example, the two-dimensional image provided by the application can be image 810.

[0137] According to an embodiment, electronic device 101 can perform focused rendering on image 810. Electronic device 101 can determine parameters for focused rendering. According to an embodiment, electronic device 101 can determine the size of a focused region 817 for focused rendering. For example, electronic device 101 can determine the size of the focused region 817 based on the location of the rendered region of the application. As an example, as the rendered region is farther away, the resolution becomes lower, so the size of the focused region 817 can be set relatively large to increase the user's visibility. For example, electronic device 101 can determine the size of the focused region 817 based on the size of the rendered region of the application. As an example, as the size of the rendered region increases, the size of the focused region 817 can also be set larger. As another example, even if the size of the rendered region increases, the size of the focused region 817 can be fixed. In other words, as the size of the rendered region increases, the focus level can decrease. Herein, the focus level can indicate the size of the focused region 817 compared to the entire image size. For example, electronic device 101 can determine the size of the focused region 817 based on the type of content applied. As an example, if the content includes a person's face, electronic device 101 can set the area including the face as the region of interest. As another example, if the content includes characters, electronic device 101 can set the area including the characters as the region of interest. If user input for scrolling the page or for zooming in on a specific area is received, electronic device 101 can also change the size of the region of interest. In this case, electronic device 101 can determine the size of the focus area 817 corresponding to the size of the region of interest. For example, electronic device 101 can determine the size of the focus area 817 based on the user's gaze data. Electronic device 101 can set the size of the focus area 817 to increase as the position of the user's gaze data gets closer to the application's rendering area.

[0138] According to an embodiment, electronic device 101 can determine the position of the focus region 817 for focused rendering. For example, electronic device 101 can determine the position of the focus region 817 based on the type of content. As an example, if the content includes a person's face, electronic device 101 can set the position of the face as the region of interest. As another example, if the content includes characters, electronic device 101 can set the area including the characters as the region of interest. If user input for scrolling the page or for zooming in on a specific area is received, electronic device 101 can also change the position of the region of interest. Electronic device 101 can determine the region of interest as the focus region 817. For example, electronic device 101 can determine the position of the focus region 817 based on the user's gaze data. Electronic device 101 can set the area within a position threshold distance from the user's gaze data as the focus region 817 in the entire image.

[0139] According to an embodiment, after determining the parameters for focused rendering, electronic device 101 can provide information about the parameters to an application. Through the application, electronic device 101 can generate a composite image of the focused rendering. Based on the execution of the application, electronic device 101 can obtain a first image 820 with high resolution for the focused region 817. Based on the execution of the application, electronic device 101 can obtain a second image 830 with low resolution for the remaining region. Based on the execution of the application, electronic device 101 can obtain a composite image by combining the first image 820 and the second image 830. Electronic device 101 can convert the composite image (e.g., a two-dimensional image) into dual image information (e.g., two two-dimensional images). For example, electronic device 101 can... Figure 6a and Figure 6b The spatialization manager 540 converts the image information corresponding to the synthesized image into dual image information corresponding to the images for both eyes. The electronic device 101 can display the rendered image corresponding to the dual image information via a display (e.g., display 250).

[0140] Figure 9 This shows an example of how the resolution changes based on the movement of an application in virtual space.

[0141] Reference Figure 9According to an embodiment, electronic device 101 can execute an application. For example, the application may be an internet browser. Electronic device 101 can display the executed application in virtual space. Electronic device 101 can display a rendered image corresponding to image information provided from the application via a display. According to an embodiment, electronic device 101 can determine the resolution of the image provided from the application based on the size and / or location of the area in virtual space where the rendered image is to be displayed (i.e., the rendering area). The resolution may not indicate the resolution of the output image converted by image processing (e.g., downsampling or upsampling) in spatialization manager 540 or virtual space manager 550, but may represent the resolution of the image provided at the application level.

[0142] Electronic device 101 may receive user input for moving a rendering area of ​​an application located at a first position 910 to a second position 920. Prior to receiving the user input, the resolution of the application's image may be a first value. For example, the first value may be determined based on a first depth 921, where the first depth 921 is the distance from a reference point of electronic device 101 (e.g., the user's position) to the first position.

[0143] Electronic device 101 (e.g., spatialization manager 540) can detect changes in the application's location. When the application's location changes, the application can be configured to query the spatialization manager 540 for display quality based on the changed location. Electronic device 101 (e.g., spatialization manager 540) can obtain the application's location information in response to user input. For example, electronic device 101 can obtain information about a second depth 922, where the second depth 922 is the distance from a reference point of electronic device 101 (e.g., the user's location) to a second location. As a non-limiting example, if resolution manager 630 is located outside of spatialization manager 540, spatialization manager 540 can provide resolution manager 630 with information about the second depth 922. Electronic device 101 (e.g., spatialization manager 540 or resolution manager 630) can change its resolution based on the second depth 922.

[0144] Electronic device 101 (e.g., spatialization manager 540 or resolution manager 630) may provide the application with resolution information (e.g., resolution information 672) in response to a query from the application. Electronic device 101 may also provide the application with resolution information indicating a changed resolution (e.g., resolution information 672).

[0145] Electronic device 101 can generate a two-dimensional image of the application by executing the application, based on the changed resolution information. Even if the rendering area of ​​the application is moved, electronic device 101 can still generate a two-dimensional image of the application by executing the application. Figure 6a and Figure 6bThe spatialization manager 540, resolution manager 630, and application 520 sequentially execute the processing procedure again to obtain a two-dimensional image based on the resolution determined at the application level. Electronic device 101 can convert image information corresponding to the generated two-dimensional image into dual-image information for both eyes (e.g., two two-dimensional images). Electronic device 101 can then display the rendered image in virtual space via a display (e.g., display 250) based on the dual-image information.

[0146] exist Figure 9 Examples of changing rendering quality as the application's rendering area moves according to user input have already been described, but embodiments of this disclosure are not limited thereto. Not only when the application's rendering area moves, but also when the distance between the user and the application's rendering area changes as the user moves, and therefore the application's rendering quality changes, can be understood as embodiments of this disclosure. For example, in response to user movement, electronic device 101 can detect changes in depth information of the application's rendering area. The application can query the spatialization manager 540 of electronic device 101 for display quality based on the change in depth information and information related to display quality (e.g., resolution information 672 indicating a changed resolution). Spatialization manager 540 (or resolution manager 630) can provide the application with information related to display quality (e.g., resolution information 672 indicating a changed resolution). Electronic device 101 can be configured to output a rendered image by the application according to the display quality.

[0147] Figure 10 The operation flow of an electronic device (e.g., electronic device 101 or wearable device 101) for controlling the resolution of an application in a virtual space is shown.

[0148] Reference Figure 10 In operation 1001, electronic device 101 (e.g., spatialization manager 540) may, in response to the execution of an application configured to provide two-dimensional images (e.g., an internet browser, gallery, calendar, settings app, or two-dimensional map), obtain depth information of the region in three-dimensional space to be rendered (hereinafter, the rendering region). Electronic device 101 (e.g., spatialization manager 540) may obtain depth information for controlling the resolution of the two-dimensional image in three-dimensional space. Here, the depth information may indicate the distance (e.g., depth, Z-coordinate, or Z-depth) between the user of electronic device 101 and the rendering region in three-dimensional virtual space. As a non-limiting example, if the position of the rendering region changes due to the execution of another application or user input, electronic device 101 may again obtain the depth information.

[0149] In operation 1003, electronic device 101 (e.g., resolution manager 630) can determine the application's resolution information based on the depth information. The application's resolution may not be indicated through separate image processing (e.g., Figure 5 The resolution of the output image converted by the spatialization manager 540 (downsampling) or the virtual spatial manager 550 (upsampling) is not the resolution of the image provided by the application at the application level.

[0150] Electronic device 101 (e.g., resolution manager 630) can determine the application's resolution information based on the application's depth information. For example, as the application's rendering area gets closer to the user, electronic device 101 can set the application's resolution to be higher. The method of setting the resolution based on the location of the rendering area can be configured differently. For example, electronic device 101 can determine the application's resolution using a function set to be inversely proportional to the depth of the rendering area. Alternatively, electronic device 101 can pre-store multiple resolution levels based on multiple distance ranges. Electronic device 101 can identify the distance range corresponding to the depth of the rendering area among multiple distance ranges. Electronic device 101 can identify the resolution level corresponding to the distance range. Alternatively, electronic device 101 can determine the application's resolution using conditions, functions, or learning models, taking into account not only the location of the rendering area but also the size of the rendering area, information about the application's content type, user input, the number of other applications being executed, the type of another application being executed, and / or the user's gaze data.

[0151] In operation 1005, electronic device 101 (e.g., spatialization manager 540) can convert image information corresponding to a two-dimensional image generated based on resolution information into dual-image information corresponding to images for both eyes. Electronic device 101 can generate a two-dimensional image based on resolution information. Electronic device 101 (e.g., spatialization manager 540) can generate image information with a two-dimensional image based on resolution information through an application. Electronic device 101 (e.g., spatialization manager 540) can convert the image information into dual-image information. The dual-image information may include first image information for the left eye (e.g., information about the two-dimensional image to be displayed on the left eye) and second image information for the right eye (e.g., information about the two-dimensional image to be displayed on the right eye).

[0152] In operation 1007, electronic device 101 can display a rendered image based on dual-image information. Electronic device 101 (e.g., virtual space manager 550) can obtain dual-image information converted by spatialization manager 540. Electronic device 101 (e.g., virtual space manager 550) can provide a rendered image corresponding to the dual-image information to a display buffer. For example, electronic device 101 can simultaneously display another application, avatar, and / or system UI (e.g., system UI 530) in virtual space along with an image of the application. Electronic device 101 can obtain the rendered image by composited with an image layer corresponding to the dual-image information and a layer for another application, avatar, and / or system UI (e.g., system UI 530). Electronic device 101 can display the rendered image via a display (e.g., a first display 250-1 or a second display 250-2).

[0153] pass Figures 5 to 10 Embodiments have been described that determine the rendering quality of an application based on depth information of the region where the application is to be rendered. However, embodiments of this disclosure are not limited thereto. According to embodiments, rendering regions that should be maintained regardless of the distance between applications can also be set. For example, in the case of advertisements or warning phrases, a certain rendering quality may be required regardless of the distance between the user and the application (e.g., being maintained at a certain value or a higher resolution). Electronic device 101 (e.g., spatialization manager 540) can be configured to not change the resolution or rendering settings even if the depth information of the application is changed, based on information about the content type and / or type of the application. In other words, electronic device 101 can set exceptions for quality changes based on the depth information of a specific type of content or a specified type of application.

[0154] In this disclosure, even when an application configured to display two-dimensional images is executed, techniques for displaying images while maintaining readability without image distortion due to resolution have been described. By controlling the resolution based on the distance to the rendering area and controlling the focus level for focused rendering, the electronic device 101 can adjust the quality of the content to be displayed on the built-in display and provide the user with content that has higher visibility. Furthermore, through a separate support module (e.g., a third-party support module 661), avatar rendering and system UI 530 can operate smoothly simultaneously within the structure of spatialization manager 540-virtual spatial manager 550. By appropriately setting the resolution and focus level, optimization can be achieved between the use of resources (e.g., GPU resources) in a trade-off relationship and the user's visibility.

[0155] The effects that can be obtained from this disclosure are not limited to those described above, and any other effects not mentioned herein will be clearly understood by those skilled in the art to which this disclosure pertains, based on the following description.

[0156] In one embodiment, an electronic device is provided. The electronic device may include: at least one display; at least one processor including processing circuitry; and a memory including one or more storage media storing instructions. When executed individually or jointly by the at least one processor, the instructions cause the electronic device to perform the following operations: in response to the execution of an application configured to provide a two-dimensional image, obtaining depth information of a region in three-dimensional space to be rendered for the application; determining resolution information of the application based on the depth information of the region; converting image information corresponding to a two-dimensional image of the application generated based on the resolution information into dual-image information corresponding to images for both eyes; and displaying the rendered image through the at least one display based on the dual-image information.

[0157] For example, when executed by at least one processor individually or jointly, the instructions may cause the electronic device to perform the following operations: receive user input for changing the position of a region in three-dimensional space to be rendered from a first position to a second position; in response to the user input, obtain information about the second position; change the resolution based on the information about the second position; convert second image information corresponding to a two-dimensional image generated based on the resolution information indicating the changed resolution into second binocular image information corresponding to an image for both eyes; and display a second rendered image through at least one display based on the second binocular image information.

[0158] For example, depth information for the region to be rendered can indicate the depth of the region to be rendered in 3D space from a reference point. The resolution, based on the resolution information, can be determined to decrease as the depth from the reference point to the region to be rendered decreases.

[0159] For example, when executed individually or jointly by at least one processor, the instructions may cause an electronic device to perform the following operations: in response to the execution of a second application configured to provide a two-dimensional image, obtaining second depth information of a second region in three-dimensional space where the second application is to be executed; determining second resolution information indicating the resolution of the second application based on the second depth information of the second region; converting second image information corresponding to a two-dimensional image of the second application generated based on the second resolution information into second binocular image information corresponding to an image for both eyes; and displaying a second rendered image via at least one display based on the second binocular image information. Where the depth indicated by the depth information is greater than the depth indicated by the second depth information, the resolution of the second application based on the second resolution information may be higher than the resolution of the application based on the resolution information.

[0160] For example, when executed individually or jointly by at least one processor, the instructions can cause the electronic device to identify the content type of the application in response to the execution of the application, and determine the application's resolution information based on information about the content type and depth information of the area. Information about the content type of the application can indicate whether characters are included in the content's area.

[0161] For example, when executed by at least one processor individually or jointly, the instructions may cause the electronic device to perform the following operations: based on the identification that the content being executed in the application has been changed, change the resolution of the indicated resolution based on information about the type of the changed content, generate modified image information corresponding to the two-dimensional image of the application based on the resolution information indicating the changed resolution, convert the modified image information into modified dual image information corresponding to the image for both eyes, and display the rendered image based on the modified dual image information.

[0162] For example, when executed individually or jointly by at least one processor, the instructions may cause the electronic device to perform the following operations: determine a focus level for focused rendering of a two-dimensional image based on depth information of the region; generate image information corresponding to the applied two-dimensional image based on resolution information and the focus level; and convert the generated image information into dual image information corresponding to images for both eyes. The focus level may indicate the size of the focused region relative to the two-dimensional image, specifically the high-resolution focused region and the remaining region with low resolution.

[0163] For example, focus level can be determined based on depth information of the area, information about the type of content in the application, and the user's gaze data. Resolution information can indicate the resolution of the focused area and the resolution of the remaining area. The user's gaze data can be obtained through the image sensor of the electronic device.

[0164] For example, when executed individually or jointly by at least one processor, the instructions may cause the electronic device to perform the following operations: determining a first value as the focus level when the location of the region is within a threshold range from the location based on the user's gaze data, and determining a second value as the focus level when the location of the region is outside the threshold range from the location based on the user's gaze data. The size of the focus area at the focus level corresponding to the first value may be larger than the size of the focus area at the focus level corresponding to the second value.

[0165] For example, when executed by at least one processor individually or jointly, the instructions may cause the electronic device to perform the following operations: determining resolution information based on depth information when the location of the region is within a threshold range from the location based on the user's gaze data, and determining the resolution information to a predetermined value when the location of the region is outside the threshold range from the location based on the user's gaze data.

[0166] For example, an application can be configured to provide image information by changing the rendering quality of a 2D image based on resolution information. The dual image information can be converted from the provided image information according to the changed rendering quality.

[0167] For example, resolution information may include resolution details.

[0168] In one embodiment, a method executed by an electronic device is provided. The method may include: in response to execution of an application configured to provide a two-dimensional image, obtaining depth information of a region in three-dimensional space to be rendered for the application; determining resolution information of the application based on the depth information of the region; converting image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes; and displaying the rendered image based on the dual image information.

[0169] For example, the method may include: receiving user input for changing the position of a region of an application to be rendered in three-dimensional space from a first position to a second position; obtaining information about the second position in response to the user input; changing the resolution based on the information about the second position; converting second image information corresponding to a two-dimensional image generated from the resolution information according to the indicated resolution change into second binocular image information corresponding to an image for both eyes; and displaying the rendered image based on the second binocular image information.

[0170] For example, depth information for the region to be rendered can indicate the depth of the region to be rendered in 3D space from a reference point. The resolution, based on the resolution information, can be determined to decrease as the depth from the reference point to the region to be rendered decreases.

[0171] For example, the method may include: in response to the execution of a second application configured to provide a two-dimensional image, obtaining second depth information for a second region in three-dimensional space where the second application is to be executed; determining second resolution information indicating the resolution of the second application based on the second depth information of the second region; converting second image information corresponding to a two-dimensional image of the second application generated based on the second resolution information into second binocular image information corresponding to images for both eyes; and displaying a second rendered image based on the second binocular image information. Where the depth indicated by the depth information is greater than the depth indicated by the second depth information, the resolution of the second application based on the second resolution information may be higher than the resolution of the application based on the resolution information.

[0172] For example, determining resolution information may include: in response to application execution, identifying the application's content type, and determining the application's resolution information based on the application's content type information and the depth information of the region. Information about the application's content type may indicate whether characters are included in the content's region.

[0173] For example, the method may include: based on identifying that the content being performed in the application has been changed, changing the resolution of the indicated resolution based on information about the type of the changed content, generating modified image information corresponding to the two-dimensional image of the application based on the resolution information indicating the changed resolution, converting the modified image information into modified dual image information corresponding to the image for both eyes, and displaying the rendered image based on the modified dual image information.

[0174] For example, generating dual-image information may include: determining a focus level for focused rendering of the 2D image based on the depth information of the region; generating image information corresponding to the applied 2D image based on resolution information and the focus level; and converting the generated image information into dual-image information for displaying images for both eyes. The focus level may indicate the size of the focused region relative to the 2D image, consisting of a high-resolution focused region and a low-resolution remaining region.

[0175] For example, focus level can be determined based on depth information of the area, information about the type of content in the application, and the user's gaze data. Resolution information can indicate the resolution of the focused area and the resolution of the remaining area. The user's gaze data can be obtained through the image sensor of the electronic device.

[0176] For example, determining the focus level may include: determining a first value as the focus level when the region's location is within a threshold range of its location based on the user's gaze data, and determining a second value as the focus level when the region's location is outside the threshold range of its location based on the user's gaze data. The size of the focused area at the focus level corresponding to the first value may be larger than the size of the focused area at the focus level corresponding to the second value.

[0177] In one embodiment, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include a memory configured to store instructions, the memory comprising one or more storage media. When executed individually or jointly by at least one processor, the instructions cause an electronic device to perform the following operations: in response to the execution of an application configured to provide a two-dimensional image, obtaining depth information of a region in three-dimensional space to be rendered for the application, determining resolution information of the application based on the depth information of the region, converting image information corresponding to a two-dimensional image of the application generated based on the resolution information into dual-image information corresponding to images for both eyes, and displaying the rendered image based on the dual-image information.

[0178] In one embodiment, an electronic device is provided. The electronic device may include: at least one display; and at least one processor, including processing circuitry. The at least one processor may be configured to: in response to execution of an application configured to provide a two-dimensional image, obtain depth information of a region in three-dimensional space to be rendered for the application, determine resolution information of the application based on the depth information of the region, convert image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes, and display the rendered image through the at least one display based on the dual image information.

[0179] In one embodiment, an electronic device is provided. The electronic device may include: a spatialization manager for obtaining spatial information of an application; a resolution manager for determining the resolution of the application; and a virtual space manager for providing an image of the application in virtual space to a display buffer. The spatialization manager may be configured to obtain depth information of a region in three-dimensional space from which the application is to be rendered. The resolution manager may be configured to determine the resolution information of the application based on the depth information of the region from which the application is to be rendered, and to provide the determined resolution information of the application to the application. The spatialization manager may be configured to convert image information generated based on the resolution information into dual-image information corresponding to images for both eyes via the application. The virtual space manager may be configured to provide a rendered image to the display buffer based on the dual-image information.

[0180] For one or more embodiments, at least one of the components described in one or more of the foregoing figures may be configured to perform one or more operations, techniques, processes, and / or methods as described in this disclosure. For example, a processor (e.g., a baseband processor) described in connection with one or more of the foregoing figures may be configured to operate according to one or more examples described in this disclosure. As another example, circuitry associated with a user equipment (UE), base station, network element, etc. (as described above in connection with one or more of the foregoing figures) may be configured to operate according to one or more examples described herein.

[0181] Unless otherwise expressly stated, any of the above embodiments may be combined with any other embodiment (or combination of embodiments). The foregoing description of one or more implementations is for illustration and explanation only, and is not intended to limit the scope of the embodiments or to exhaustively describe the precise forms disclosed. Modifications and alterations are possible in accordance with the above teachings, or may be obtained from practice of various embodiments.

[0182] For one or more embodiments, at least one of the components described in one or more of the foregoing figures may be configured to perform one or more operations, techniques, processes, and / or methods as described in this disclosure. For example, a processor (e.g., a baseband processor) described in connection with one or more of the foregoing figures may be configured to operate according to one or more examples described in this disclosure. As another example, circuitry associated with a user equipment (UE), base station, network element, etc. (as described above in connection with one or more of the foregoing figures) may be configured to operate according to one or more examples described herein.

[0183] Unless otherwise expressly stated, any of the above embodiments may be combined with any other embodiment (or combination of embodiments). The foregoing description of one or more implementations is for illustration and explanation only, and is not intended to limit the scope of the embodiments or to exhaustively describe the precise forms disclosed. Modifications and alterations are possible in accordance with the above teachings, or may be obtained from practice of various embodiments.

[0184] The electronic device according to various embodiments can be one of a variety of types of electronic devices. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. According to embodiments of this disclosure, the electronic device is not limited to those described above.

[0185] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to the specific embodiments, but rather to include various changes, equivalents, or substitutions to the respective embodiments. In the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that nouns in the singular form corresponding to terms may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any one or all possible combinations of the items enumerated together with the corresponding phrase among the plurality of phrases. As used herein, terms such as “first” and “second” or “first” and “second” may be used to simply distinguish one component from another and do not limit the components in other respects (e.g., importance or order). It will be understood that, whether the terms “operably” or “communically” are used or not, if an element (e.g., a first element) is referred to as “coupled to another element (e.g., a second element)” or “connected to another element (e.g., a second element)”, it means that the element can be directly (e.g., wiredly) coupled to the other element, wirelessly coupled to the other element, or coupled to the other element via a third element.

[0186] As used in connection with various embodiments of this disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as "logic," "logic block," "part," or "circuit." A module may be a single integrated component adapted to perform one or more functions, or the smallest unit or part of such a single integrated component. For example, according to embodiments, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0187] The various embodiments set forth herein can be implemented as software (e.g., program 140) containing one or more instructions readable by a machine (e.g., electronic device 101) stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, under the control of a processor, the processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke and execute at least one of the one or more instructions stored in the storage medium, with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. Machine-readable storage media may be provided in the form of non-transitory storage media. The term "non-transitory" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is temporarily stored in the storage medium.

[0188] According to embodiments, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM The computer program product may be published online (e.g., downloaded or uploaded), or may be distributed directly between two user devices (e.g., smartphones) (e.g., downloaded or uploaded). If published online, at least a portion of the computer program product may be temporarily generated, or at least a portion of the computer program product may be temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a forwarding server).

[0189] According to various embodiments, each of the above components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be separately disposed in different components. According to various embodiments, one or more of the above components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding component of the multiple components performed one or more functions before integration. According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be run in a different order or omitted, or one or more other operations may be added.

Claims

1. An electronic device comprising: At least one display; At least one processor, including processing circuitry; as well as The memory includes one or more storage media storing instructions that, when executed individually or jointly by the at least one processor, cause the electronic device to perform the following operations: In response to the execution of an application configured to provide two-dimensional images, depth information of the region in three-dimensional space to be rendered by the application is obtained; Based on the depth information of the region, the resolution information of the application is determined; The image information corresponding to the two-dimensional image of the application generated based on the resolution information is converted into dual image information corresponding to the image for both eyes; as well as Based on the dual image information, the rendered image is displayed on the at least one display.

2. The electronic device as claimed in claim 1, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device causes to perform the following operations: Receive user input for changing the position of the region in the three-dimensional space to be rendered from a first position to a second position; In response to the user input, information about the second location is obtained; The resolution is adjusted based on information about the second position; The resolution information generated according to the indicated resolution change is converted into second image information corresponding to the two-dimensional image, which is then converted into second binocular image information corresponding to the images for both eyes. as well as Based on the second dual-image information, a second rendered image is displayed on the at least one display.

3. The electronic device as claimed in claim 1, in, The depth information of the region to be rendered in the three-dimensional space indicates the depth from the reference point to the region to be rendered in the three-dimensional space. The resolution, based on the resolution information, is determined to decrease as the depth from the reference point to the area to be rendered decreases.

4. The electronic device as claimed in claim 1, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device causes to perform the following operations: In response to the execution of a second application configured to provide a two-dimensional image, second depth information of a second region in the three-dimensional space where the second application is to be executed is obtained; Based on the second depth information of the second region, second resolution information indicating the resolution of the second application is determined; The second image information corresponding to the two-dimensional image of the second application generated based on the second resolution information is converted into second dual image information corresponding to the image for both eyes; as well as Based on the second dual-image information, a second rendered image is displayed through the at least one display. Where the depth indicated by the depth information is greater than the depth indicated by the second depth information, the resolution of the second application based on the second resolution information is higher than the resolution of the application based on the resolution information.

5. The electronic device as claimed in claim 1, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device causes to perform the following operations: In response to the execution of the application, identify the type of content of the application; as well as Based on information about the type of content of the application and the depth information of the region, resolution information indicating the resolution of the application is determined, and The information regarding the type of content of the application indicates whether the character is included in the area of ​​the content.

6. The electronic device as claimed in claim 1, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device causes to perform the following operations: Based on the identification that the content being executed in the application has been changed, the resolution is adjusted based on information about the type of the changed content. Based on the resolution information indicating a change in resolution, modified image information corresponding to the two-dimensional image of the application is generated; The modified image information is converted into modified dual image information corresponding to the images for both eyes; as well as Based on the modified dual-image information, the rendered image is displayed on the at least one display.

7. The electronic device as claimed in claim 1, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device causes to perform the following operations: Based on the depth information of the region, a focus level is determined for focus rendering of the two-dimensional image; Based on the resolution information and the focus level, image information corresponding to the two-dimensional image of the application is generated; as well as The generated image information is converted into dual-image information corresponding to images for both eyes, and The focus level indicates the size of the focus area relative to the high-resolution focus area and the low-resolution remaining area of ​​the two-dimensional image.

8. The electronic device as claimed in claim 7, in, The focus level is determined based on the depth information of the region, information about the type of content on the application, and the user's gaze data. The resolution information indicates the resolution of the focused area and the resolution of the remaining area, and The user's gaze data is obtained through the image sensor of the electronic device.

9. The electronic device as claimed in claim 7, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device causes to perform the following operations: If the location of the area is within a threshold range of the location based on the user's gaze data, a first value is determined as the focus level. If the location of the area is outside the threshold range based on the user's gaze data, the second value is determined as the focus level, and Wherein, the size of the focused area at the focus level corresponding to the first value is greater than the size of the focused area at the focus level corresponding to the second value.

10. The electronic device of claim 1, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device causes to perform the following operations: If the location of the region is within a threshold range from the location based on the user's gaze data, the resolution information is determined based on the depth indicated by the depth information. If the location of the area is outside the threshold range of the location based on the user's gaze data, the resolution information is determined to be a predetermined value.

11. The electronic device as claimed in claim 1, in, The application is configured to provide the image information by changing the rendering quality of the 2D image based on the resolution information, and The dual image information is converted from the provided image information based on the changed rendering quality.

12. The electronic device as claimed in claim 1, wherein, The resolution information includes a scaling factor.

13. A method performed by an electronic device, the method comprising: In response to the execution of an application configured to provide two-dimensional images, depth information of the region in three-dimensional space to be rendered by the application is obtained; Based on the depth information of the region, the resolution information of the application is determined; The image information generated based on the resolution information and corresponding to the two-dimensional image of the application is converted into dual image information corresponding to the image for both eyes; as well as The rendered image is displayed based on the dual image information.

14. The method of claim 13, further comprising: Receive user input for changing the position of the region in the three-dimensional space to be rendered from a first position to a second position; Information about the second location is obtained in response to the user input; The resolution is adjusted based on information about the second position; The resolution information generated according to the indicated resolution change is converted into second image information corresponding to the two-dimensional image, which is then converted into second binocular image information corresponding to the images for both eyes. as well as The second rendered image is displayed based on the second dual-image information.

15. The method as described in claim 13, in, The depth information of the region in 3D space to be rendered indicates the depth from the reference point to the region in 3D space to be rendered. The resolution determined based on the resolution information is set to decrease as the depth from the reference point to the area to be rendered decreases.

Images