Electronic device, method, and computer-readable medium for rendering images

Abstract

In embodiments, an electronic device is provided. The electronic device may comprise: at least one display; at least one processor including a processing circuit; and a memory storing instructions and including one or more storage media. The instructions, when executed individually or collectively by the at least one processor, may instruct the electronic device to: obtain depth information about an area in which an application is to be rendered in a three-dimensional space in response to execution of the application configured to provide a two-dimensional image; determine resolution information for the application according to the depth information about the area; convert image information corresponding to the two-dimensional image of the application generated according to the resolution information into dual image information corresponding to images for both eyes; and display a rendered image on the at least one display on the basis of the dual image information.

Description

Electronic device, method, and computer-readable medium for rendering images

[0001] The present disclosure relates to an electronic device, method, and computer-readable medium for rendering an image.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices can provide AR services to users by utilizing virtual objects corresponding to the user.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] In embodiments, an electronic device is provided. The electronic device may include at least one display, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device, in response to execution of an application configured to provide a two-dimensional image, to obtain depth information for an area in a three-dimensional space where the application is to be rendered, determine resolution information for the application based on the depth information for the area, convert image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes, and display a rendered image through the at least one display based on the dual image information.

[0005] In embodiments, a method performed by an electronic device is provided. The method may include, in response to execution of an application configured to provide a two-dimensional image, acquiring depth information for an area in a three-dimensional space where the application is to be rendered; determining resolution information for the application based on the depth information for the area; converting image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes; and displaying a rendered image based on the dual image information.

[0006] In embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include a memory configured to store instructions, the memory including one or more storage media. The instructions, when individually or collectively executed by at least one processor, may cause an electronic device to, in response to execution of an application configured to provide a two-dimensional image, obtain depth information for an area in three-dimensional space where the application is to be rendered, determine resolution information for the application based on the depth information for the area, convert image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes, and display a rendered image based on the dual image information.

[0007] In embodiments, an electronic device is provided. The electronic device may include at least one display and at least one processor including a processing circuit. The at least one processor may be configured to, in response to execution of an application configured to provide a two-dimensional image, obtain depth information for an area in a three-dimensional space where the application is to be rendered, determine resolution information for the application based on the depth information for the area, convert image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes, and display a rendered image through the at least one display based on the dual image information.

[0008] In embodiments, an electronic device is provided. The electronic device may include a spatialization manager for obtaining spatial information about an application; a resolution manager for determining a resolution for the application; and a virtual space manager for providing an image for the application in a virtual space to a display buffer. The spatialization manager may be configured to obtain depth information about an area in which the application is to be rendered in a three-dimensional space. The resolution manager may be configured to determine resolution information for the application based on the depth information about the area in which the application is to be rendered, and provide the determined resolution information for the application to the application. The spatialization manager may be configured to convert image information generated by the application based on the resolution information into dual image information corresponding to images for both eyes. The virtual space manager may be configured to provide rendered images based on the dual image information to the display buffer.

[0009] Figure 1 is a block diagram of an electronic device within a network environment.

[0010] Figure 2a shows an example of a perspective view of a wearable device.

[0011] FIG. 2b illustrates an example of one or more hardware elements arranged within a wearable device.

[0012] Figures 3a and 3b show an example of the appearance of a wearable device.

[0013] Figure 4 shows an example of a block diagram of a wearable device.

[0014] Fig. 5 shows an example of a block diagram of an electronic device for displaying an image in a virtual space.

[0015] Figures 6a and 6b illustrate examples of block diagrams of electronic devices for controlling the resolution of an image in virtual space.

[0016] Figure 7 shows examples of applications displayed in virtual space.

[0017] Figure 8 shows examples of images for foveated rendering.

[0018] Figure 9 shows an example of resolution change according to movement of an application in virtual space.

[0019] Figure 10 shows the operation flow of an electronic device for controlling the resolution of an application in a virtual space.

[0020] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0021] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0022] In the following description, terms referring to space (e.g., plane, object, shape, surface, figure, solid figure, area, occupied area, location, depth, distance), terms referring to distance (e.g., location, distance, depth, depth information, distance information, distance value, depth value, location information, location data, depth data), terms referring to applications (e.g., app, application, program, app activity, application activity, activity, function), terms referring to values ​​(e.g., threshold, reference value, reference area, reference range, level, threshold, range, value, area), terms for operation states (e.g., step, operation, procedure), terms referring to network entities, terms referring to components of a device, etc., are examples for convenience of description. Therefore, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings may be used.

[0023] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled, but this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." A condition described as "more than" may be replaced with "more than," a condition described as "less than" may be replaced with "less than," and a condition described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of elements from A (including A) to B (including B). hereinafter, "C" and / or "D" mean at least one of "C" or "D," that is, including {"C", "D", "C" and "D"}.

[0024] Figure 1 is a block diagram of an electronic device within a network environment.

[0025] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0026] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0027] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0028] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0029] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0030] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0031] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0032] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0033] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0034] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0035] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0036] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0037] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0038] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0039] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0040] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0041] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0042] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0043] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0044] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0045] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0046] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0047] In embodiments of the present disclosure, an electronic device (e.g., electronic device (101) of FIG. 1) for displaying an image in a virtual space may be a wearable device. The wearable device (101) may include a head-mounted display (HMD) that is wearable on a user's head. The wearable device (101) may be referred to as a head-mounted device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of a hardware configuration included in the wearable device (101) is exemplarily described with reference to FIG. 4. An example of the structure of a wearable device (101) that can be worn on the head of a user (110) is described with reference to FIGS. 2A, 2B, 3A, and / or 3B. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) for attaching to the head of a user to form an HMD.

[0048] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) may see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0049] In one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include a housing that covers the eyes of the user (110). The wearable device (101), in this state, may include a display disposed on a first side of the housing facing the eyes. The wearable device (101) may include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (101) may acquire images and / or videos representing ambient light. The wearable device (101) can output the image and / or video within the display disposed on the first surface, thereby allowing the user (110) to recognize the ambient light through the display. The displaying area (or displaying region) (or active area or active region) of the display disposed on the first surface can be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object into the image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with a real object recognized by the ambient light.

[0050] According to one embodiment, the wearable device (101) can identify or recognize a position (or location) and / or direction (or orientation) of the wearable device (101) based on an image (and / or video) obtained or acquired using a camera. The wearable device (101) can obtain information about the external space using one or more cameras and / or one or more sensors. The information can include a geographic location (e.g., global positioning system (GPS) coordinates) of the external space identified from one or more sensors. The information can include images and / or videos of the external space identified from one or more cameras. The wearable device (101) can perform object recognition on the images and / or videos to identify external objects included in the external space from the images and / or videos.

[0051] Below, an example of a hardware configuration of a wearable device (101) is described with reference to FIGS. 2a, 2b, 3a, 3b, and 4.

[0052] FIG. 2A illustrates an example of a perspective view of a wearable device. FIG. 2B illustrates an example of one or more hardware components arranged within the wearable device. According to one embodiment, the wearable device (101) may have a form of glasses that can be worn on a body part (e.g., head) of a user. The wearable device (101) of FIGS. 2A and 2B may be an example of the wearable device (101) of FIG. 1. The wearable device (101) may include a head-mounted display (HMD). For example, the housing of the wearable device (101) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (101) may include one or more straps capable of being twined around the user's head, and / or one or more temples attachable to the ears of the head.

[0053] Referring to FIG. 2A, according to one embodiment, a wearable device (101) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0054] According to one embodiment, a wearable device (101) can be worn on a part of a user's body. The wearable device (101) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (101). For example, the wearable device (101) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0055] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0056] Referring to FIG. 2B, at least one display (250) can provide visual information transmitted from external light to the user through a lens included in the at least one display (250), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When a user wears the wearable device (101), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0057] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (101) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0058] The wearable device (101) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (101) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (101) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (101) can view an image displayed on at least one display (250).

[0059] According to one embodiment, the frame (200) may be formed as a physical structure that allows the wearable device (101) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that, when the user wears the wearable device (101), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0060] Referring to FIG. 2A, the frame (200) may include a region (220) that is in contact with at least a portion of a user's body when the user wears the wearable device (101). For example, the region (220) of the frame (200) that is in contact with a portion of the user's body may include a region that is in contact with a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (101) makes contact with. According to one embodiment, the frame (200) may include a nose pad (210) that is in contact with a portion of the user's body. When the wearable device (101) is worn by the user, the nose pad (210) may be in contact with a portion of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another portion of the user's body that is distinct from the portion of the user's body.

[0061] For example, the frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (101) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200).

[0062] According to one embodiment, the wearable device (101) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (200).

[0063] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (101) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B, but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B. When the number of microphones (265) included in the wearable device (101) is two or more, the wearable device (101) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (200).

[0064] According to one embodiment, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or may be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (101) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0065] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (101). For example, the wearable device (101) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (101) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (101) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other. The wearable device (101) can obtain an image having visual quality of a first area matching the user's gaze and visual quality of a second area using foveated rendering.For example, if the wearable device (101) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1). An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0066] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (101) can compensate for depth information (e.g., the distance between the wearable device (101) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (260-4). The wearable device (101) can perform object recognition through an image acquired using the capturing camera (260-4). The wearable device (101) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capturing camera (260-4). The wearable device (101) can perform a pass-through function to display an image acquired through the capturing camera (260-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the capturing camera (260-4) can be disposed on a bridge (203) disposed between the first rim (201) and the second rim (202).

[0067] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing a wearable device (101), thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (101) looks straight ahead, the wearable device (101) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (101) is positioned.

[0068] The motion recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The motion recognition camera (260-2, 260-3) can recognize the user's motion (gesture recognition), obtain a signal corresponding to the motion, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the motion, and perform a designated function based on the identification. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor can perform a gesture recognition function and / or an object tracking function using the motion recognition camera (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) may be positioned on the first rim (201) and / or the second rim (202).

[0069] The camera (260) included in the wearable device (101) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (101) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (101) can identify an external object based on a sensor for identifying the distance between the wearable device (101) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (101) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (101).

[0070] Although not shown, in one embodiment, the wearable device (101) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame (200) and the hinge units (206, 207).

[0071] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (101). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0072] The antenna module (275) can transmit signals or power to the outside of the wearable device (101), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0073] The speaker (255) can output an acoustic signal to the outside of the wearable device (101). The acoustic output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (101). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus positioned adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus positioned adjacent to the user's right ear.

[0074] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (101) to the user. For example, when the wearable device (101) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0075] Referring to FIG. 2B, according to one embodiment, a wearable device (101) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (101) (e.g., hardwares illustrated by different blocks in FIG. 4) may be disposed on the PCB (290). The wearable device (101) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0076] According to one embodiment, a wearable device (101) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (101) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (101). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (101) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (101) based on the IMU.

[0077] FIGS. 3A and 3B illustrate an example of an exterior appearance of a wearable device (e.g., a wearable device (101)). The wearable device (101) of FIGS. 3A and 3B may be an example of the wearable device (101) of FIG. 1 . According to one embodiment, an example of an exterior appearance of a first side (310) of a housing of the wearable device (101) is illustrated in FIG. 3A , and an example of an exterior appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B .

[0078] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (101) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (101) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (101) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0079] According to one embodiment, the wearable device (101) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (101) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (101) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (101) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar expressing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (101).

[0080] Referring to FIG. 3B, a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor (330)) for obtaining information related to the external environment of the wearable device (101) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0081] For example, using cameras (260-11, 260-12), the wearable device (101) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be placed on the second face (320) of the wearable device (101) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be placed on the second face (320) of the wearable device (101) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0082] According to one embodiment, the wearable device (101) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (101) and an external object. Using the depth sensor (330), the wearable device (101) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (101). Although not illustrated, a microphone may be disposed on the second face (320) of the wearable device (101) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0083] Hereinafter, with reference to FIG. 4, the hardware or software configuration of the wearable device (101) is described.

[0084] Fig. 4 illustrates an example of a block diagram of a wearable device (e.g., a wearable device (101)). The wearable device (101) of Fig. 4 may be an example of the electronic device (101) of Fig. 1 or the wearable devices (101) of Figs. 2A to 3B.

[0085] Referring to FIG. 4, a wearable device (101) according to one embodiment may include a processor (410), a memory (415), a display (250) (e.g., the first display (250-1) and / or the second display (250-2) of FIGS. 2A, 2B, 3A, and 3B), and / or a sensor (420) (e.g., the image sensor (421) and / or the motion sensor (422)). The processor (410), the memory (415), the display (250), and / or the sensor (420) may be electrically and / or operatively connected to each other by electronic components such as a communication bus (402). In the present disclosure, the operative connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (101) is not limited to those illustrated in FIG. 4. For example, the wearable device (101) may include only some of the electronic components illustrated in FIG. 4.

[0086] A processor (410) of a wearable device (101) according to one embodiment may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. The processor (410) may have a multi-core processor structure such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (410) may include a structure (e.g., a big-little structure) based on a plurality of core circuits that are distinguished by power consumption, clock, and / or calculation amount per unit time. In one embodiment comprising a processor (410) having a multi-core processor architecture, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (410).

[0087] According to one embodiment, the memory (415) of the wearable device (101) may include electronic components for storing data and / or instructions input to and / or output from the processor (410). The memory (415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC). In one embodiment, the memory (415) may be referred to as storage.

[0088] In one embodiment, the display (250) of the wearable device (101) can output visualized information to the user of the wearable device (101). The display (250), which is arranged in front of the eyes of the user wearing the wearable device (101), can be arranged on at least a part of the housing of the wearable device (101) (e.g., the first display (250-1) and / or the second display (250-2) of FIGS. 2A, 2B, 3A, and 3B). For example, the display (250) can be controlled by a processor (410) including circuits such as a CPU (411), a GPU (graphics processing unit) (412), and / or a DPU (display processing unit) (413), to output visualized information to the user. The display (250) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (250) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). The embodiment is not limited thereto, and for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (250) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (250) may be referred to as a display panel and / or a display module. The pixels included in the display (250) may be arranged to face either of the user's eyes when the wearable device (101) is worn by the user.For example, the display (250) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0089] In one embodiment, the sensor (420) of the wearable device (101) may generate electrical information that may be processed by the processor (410) and / or the memory (415) from non-electronic information related to the wearable device (101). For example, the sensor (420) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (101). In addition to the GPS method, the sensor (420) may generate information indicating the geographic location of the wearable device (101) based on a global navigation satellite system (GNSS) such as, for example, Galileo or Beidou (compass). The information may be stored in the memory (415), processed by the processor (410), and / or transmitted to another electronic device distinct from the wearable device (101) via a communication circuit.

[0090] Referring to FIG. 4, an image sensor (421) and / or a motion sensor (422) are illustrated as examples of a sensor (420) included in a wearable device (101). The sensor (420) may include one or more optical sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate electrical signals representing the color and / or brightness of light. The image sensor (421) may be referred to as a camera. A plurality of optical sensors included in the image sensor (421) may be arranged in the form of a two-dimensional grid (4-dimensional array). The image sensor (421) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional grid. For example, photographic data captured using the image sensor (421) may mean one (a) two-dimensional frame data acquired from the image sensor (421). For example, video data captured using the image sensor (421) may mean a sequence of a plurality of two-dimensional frame data acquired from the image sensor (421) according to a frame rate. The image sensor (421) may be arranged toward the direction in which the image sensor (421) receives light and may further include a flash light for outputting light toward the direction.

[0091] According to one embodiment, the wearable device (101) may include a plurality of image sensors, as an example of an image sensor (421), arranged in different directions. As described above with reference to FIGS. 2A, 2B, 3A, and 3B, the plurality of image sensors may include gaze tracking cameras (e.g., gaze tracking camera 260-1 of FIGS. 2B and 3A) configured to be arranged toward the eyes of a user wearing the wearable device (101). The plurality of image sensors may include outward cameras. The processor (410) may identify the direction of the user's gaze using images and / or videos acquired from the gaze tracking cameras. The gaze tracking cameras may include infrared (IR) sensors. The gaze tracking cameras may be referred to as eye sensors and / or eye trackers.

[0092] For example, the external camera may be positioned facing the front of a user wearing the wearable device (101) (e.g., in a direction that both eyes may face). The wearable device (101) may include multiple external cameras. The embodiment is not limited thereto, and the external camera may be positioned facing an external space. Using images and / or videos acquired from the external cameras, the processor (410) may identify external objects. For example, based on images and / or videos acquired from the external cameras, the processor (410) may identify the position, shape, and / or gesture (e.g., hand gesture) of a hand of a user wearing the wearable device (101). Using images and / or videos of the external environment acquired from the external cameras, the processor (410) may recognize or track one or more objects within the external environment.

[0093] In one embodiment, the motion sensor (422) may output electrical signals representing gravitational accelerations, accelerations, and / or angular velocities of a plurality of axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other and based on a designated origin within the wearable device (101) and / or the motion sensor (422). For example, the processor (410) may repeatedly receive or acquire sensor data including accelerations, angular velocities, and / or magnitudes of magnetic fields of the plurality of axes from the motion sensor (422) based on a designated period (e.g., 1 millisecond). In one embodiment, the motion sensor (422) may be referred to as an inertial measurement unit (IMU). The sensor (420) included in the wearable device (101) is not limited to those described above, and may include a grip sensor, a proximity sensor, a heart rate sensor, a fingerprint sensor, an ambient light sensor, and / or a ToF sensor. Using the motion sensor (422), the processor (410) can detect motion of the wearable device (101) (e.g., motion of the wearable device (101) caused by a user wearing the wearable device (101).

[0094] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (410) of the wearable device (101) may be stored in the memory (415) of the wearable device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (101) and / or the processor (410) may perform at least one of the operations of FIGS. 6A, 6B, 7, 8, 9, and 10 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, a program, and / or a software application is executed. Hereinafter, the fact that a software application is installed in a wearable device (101) may mean that one or more instructions provided in the form of a software application (or package) are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (101)). For example, the application may include a program and / or a library related to a service provided to a user.

[0095] Referring to FIG. 4, programs installed in the wearable device (101) may be included in any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (480), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (250), and / or the sensor (420)) of the wearable device (101) may be included in the hardware abstraction layer (480). The framework layer (450) may be referred to as an XR framework layer from the perspective of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 4 may be logically (or for convenience of explanation) separated, and may not mean that the address space of the memory (415) is separated by the layers.

[0096] For example, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (480) and / or the application layer (440) (e.g., a position tracker (471), a space recognizer (472), a gesture tracker (473), and / or an eye-gaze tracker (474)) may be included. The programs included in the framework layer (450) may provide an API (application programming interface) that is executable (or callable) based on other programs.

[0097] For example, the application layer (440) may include a program designed to target users of the wearable device (101). As an example of programs included in the application layer (440), an extended reality (XR) system user interface (UI) (441) and / or an XR application (442) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (440) may call an API to cause execution of functions supported by programs included in the framework layer (450).

[0098] For example, the wearable device (101) may display one or more visual objects on the display (250) for performing interaction with the user based on the execution of the XR system UI (441). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user with functions available within a virtual space based on the execution of the XR system UI (441).

[0099] Referring to FIG. 4, a lightweight renderer (443) and / or an XR plug-in (444) are illustrated to be included within the XR system UI (441), but are not limited thereto. For example, based on the XR system UI (441), the processor (410) may execute a lightweight renderer (443) and / or an XR plug-in (444) within the framework layer (450).

[0100] For example, the wearable device (101) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (443). The lightweight renderer (443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (443) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (444). The XR plugin (444) may be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0101] For example, the wearable device (101) may display a screen representing at least a portion of a virtual space on the display (250) based on the execution of the XR application (442). The XR plug-in (444-1) included in the XR application (442) may include instructions that support functions similar to those of the XR plug-in (444) of the XR system UI (441). Descriptions of the XR plug-in (444-1) that overlap with those of the XR plug-in (444) may be omitted. The wearable device (101) may cause the execution of the virtual space manager (451) based on the execution of the XR application (442).

[0102] For example, the wearable device (101) may display an image on the display (250) in a virtual space based on the execution of the application (445). The application (445) may be configured to output image information for displaying a two-dimensional image. The wearable device (101) may cause the execution of the virtual space manager (451) based on the execution of the application (445). The wearable device (101) may generate dual image information to display the two-dimensional image in a three-dimensional virtual space based on the execution of the application (445). Here, the dual image information may include first image information for the left eye and second image information for the right eye in consideration of binocular parallax. In order to display the two-dimensional image in the three-dimensional virtual space, the wearable device (101) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0103] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the virtual space manager (451). For example, the virtual space manager (451) may include a platform for supporting the virtual space service. The wearable device (101) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (430) based on the execution of the virtual space manager (451), and may display at least a portion of the virtual space on the display (250). The virtual space manager (451) may be referred to as a composition presentation manager (CPM).

[0104] For example, the virtual space manager (451) may include a runtime service (452). As an example, the runtime service (452) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (452). As an example, the wearable device (101) may perform rendering for a virtual space service for the user based on the execution of the runtime service (452). For example, a function related to a virtual space, executable by the application layer (440), may be supported based on the execution of the runtime service (452).

[0105] For example, the virtual space manager (451) may include a pass-through manager (453). Based on the execution of the pass-through manager (453), the wearable device (101) may display a screen representing a virtual space (e.g., screen (120) of FIG. 1) on the display (250), while displaying an image and / or video representing an actual space acquired through an external camera by overlaying it on at least a portion of the screen.

[0106] For example, the virtual space manager (451) may include an input manager (454). The wearable device (101) may identify data (e.g., sensor data) obtained by executing one or more programs included in the recognition service layer (470) based on the execution of the input manager (454). The wearable device (101) may use the obtained data to identify user input related to the wearable device (101). The user input may be related to a motion (e.g., a hand gesture), gaze, and / or speech of the user identified by a sensor (420) (e.g., an image sensor (430) such as an external camera). The user input may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0107] For example, the perception abstract layer (460) can be used for data exchange between the virtual space manager (451) and the perception service layer (470). From the perspective of being used for data exchange between the virtual space manager (451) and the perception service layer (470), the perception abstract layer (460) can be referred to as an interface. For example, the perception abstract layer (460) can be referenced as OpenPX. The perception abstract layer (460) can be used for a perception client and a perception service.

[0108] According to one embodiment, the recognition service layer (470) may include one or more programs for processing data acquired from the sensor (420). The one or more programs may include at least one of a position tracker (471), a space recognizer (472), a gesture tracker (473), and / or an eye tracker (474). The type and / or number of the one or more programs included in the recognition service layer (470) are not limited to those illustrated in FIG. 4.

[0109] For example, the wearable device (101) can identify the pose of the wearable device (101) using the sensor (430) based on the execution of the position tracker (471). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using an external camera (e.g., an image sensor (421)) and / or an IMU (e.g., a motion sensor (422) including a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (471). The position tracker (471) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0110] For example, the wearable device (101) may obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of the space recognizer (472). The wearable device (101) may reproduce the surrounding environment of the wearable device (101) in three dimensions using data obtained using an external camera (e.g., an image sensor (421)) based on the execution of the space recognizer (472). The wearable device (101) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (101) reproduced in three dimensions based on the execution of the space recognizer (472). The space recognizer (472) may be referred to as a scene understanding (SU) module (or a scene recognition program).

[0111] For example, the wearable device (101) may identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (101) based on the execution of the gesture tracker (473). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user using data acquired from an external camera (e.g., an image sensor (421)) based on the execution of the gesture tracker (473). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using an external camera based on the execution of the gesture tracker (473). The gesture tracker (473) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0112] For example, the wearable device (101) may identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (474). As an example, the wearable device (101) may identify eye movements of the user using data acquired from a gaze tracking camera (e.g., an image sensor (421)) based on the execution of the gaze tracker (474). The gaze tracker (474) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0113] Referring to FIG. 4, examples of a processor (410) include a CPU (411), a GPU (graphics processing unit) (412), and / or a DPU (display processing unit) (413). A renderer (490) may include instructions for rendering images in a three-dimensional virtual space. A processor (410) (e.g., DPU (413)) executing the renderer (490) may obtain at least one image to be at least partially displayed in a display area of ​​a display (250) from a software application (e.g., a software application executed by the CPU (411) and / or GPU (412)). For example, a processor (410) executing the renderer (490) may determine a location of an area in which an application (e.g., an XR application (242), an application (245)) is to be rendered. The processor (410) executing the renderer (490) can generate an image of the application to be displayed on the display (250). The renderer (490) can synthesize images to generate a composite image to be displayed on the display (250).

[0114] For example, the processor (410) executing the renderer (490) can divide the display area of ​​the display (250) into a foveated portion (or may be referred to as the foveated area) and a peripheral portion (or may be referred to as the residual area) using the gaze position calculated using the position tracker (471) and / or the gaze tracker (474). For example, the processor (410) detecting the coordinate values ​​of the gaze position can determine the portion of the display area including the coordinate values ​​as the foveated area. The DPU (413) executing the renderer (490) can obtain at least one image corresponding to each of the foveated area and the residual area, and having a size smaller than the size of the entire display area of ​​the display (250) or a resolution smaller than the resolution of the display area.

[0115] The processor (410) executing the renderer (490) may obtain or generate a composite image to be displayed on the display (250) by synthesizing an image corresponding to the foveated area and an image corresponding to the peripheral area. For example, the processor (410) may perform upscaling to enlarge the image corresponding to the peripheral area to the size of the entire display area of ​​the display (250). On the enlarged image, the processor (410) may combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (250). Along the boundary line of the image corresponding to the foveated area, the processor (410) may apply a visual effect, such as blur, to blend the enlarged image and the image corresponding to the foveated area.

[0116] Fig. 5 shows an example of a block diagram of an electronic device (e.g., electronic device (101), wearable device (101)) for displaying an image in a virtual space. In Fig. 5, an example of executing a plurality of programs / instructions for displaying an image in a virtual space is described. The plurality of programs / instructions may all be executed in one processor (e.g., AP) or may be executed by a plurality of processors (e.g., AP, GPU (graphics processing unit), NPU (neural processing unit)). The meaning of being executed by the plurality of processors means that some programs / instructions may be executed by a first processor and other some programs / instructions may be executed by a second processor different from the first processor.

[0117] Referring to FIG. 5, the electronic device (101) may execute a virtual space manager (550) (e.g., the virtual space manager (451) of FIG. 4, CPM) to render an image in a virtual space. For the virtual space manager (550), at least some of the descriptions of the virtual space manager (451) of FIG. 4 may be referred to. The virtual space manager (550) may include a platform for supporting a virtual space service. The virtual space manager (550) may include a runtime service (551) (e.g., OpenXR Runtime), a panel renderer (552) (e.g., 2D Panel Render), and an XR compositor (553). The electronic device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (551). For the runtime service (551), at least some of the descriptions of the runtime service (452) of FIG. 4 may be referred to. The electronic device (101) may display at least one image (video) on a panel (e.g., a 2D panel) to implement a virtual space through the display based on the execution of the panel rendering (552). For example, the electronic device (101) may display a rendering image corresponding to RGB information (566) for the panel from the spatialization manager (540) described below through the display (e.g., the display (250)). The electronic device (101) may synthesize an image of an actual area captured by a camera in a virtual space (hereinafter, a pass-through image) with a virtual area image based on the execution of the XR compositor (553). For example, the electronic device (101) may generate a composite image by merging the pass-through image and the virtual area image based on the execution of the XR compositor (553). The electronic device (101) can transmit the generated composite image to a display buffer so that the composite image is displayed.The electronic device (101) can identify a virtual space through a virtual space manager (550) and display at least a portion of the virtual space on the display (250). The virtual space manager (550) may be referred to as a CPM. The electronic device (101) can execute the virtual space manager (550) to render an image corresponding to at least a portion of the virtual space.

[0118] According to one embodiment, the electronic device (101) may execute a spatialization manager (540). The spatialization manager (540) may perform processes for displaying an image in a three-dimensional virtual space. The electronic device (101) may perform preprocessing based on the execution of the spatialization manager (540) so that the image can be rendered in a three-dimensional virtual space through the virtual space manager (550). For example, the electronic device (101) may perform at least some of the functions of the renderer (490) of FIG. 4 based on the execution of the spatialization manager (540). The electronic device (101) may process image information provided by an application (e.g., an XR application (510), an application (520) that provides a general 2D screen other than XR, and an application that provides a system UI (530)) based on the execution of the spatialization manager (540). A spatialization manager (540) (e.g., Space Flinger) may include a system scene manager (541) (e.g., System scene), an input manager (542) (e.g., Input Routing), and a lightweight rendering engine (543) (e.g., Impress Engine). The system scene manager (541) may be executed to display a system UI (530). System UI-related information (564) may be transmitted to the system scene manager (541) from a program (e.g., API) that provides the system UI (530). The system UI-related information (564) may be obtained through a spatializer API and / or a same-process private API. The spatialization manager (540) may determine the layout (e.g., location, display order) of the screen of the system UI (530) in a three-dimensional space through pre-allocated resources. The system screen manager (541) can transmit image information (567) for rendering the screen of the system UI (530) to the virtual space manager (550) according to the above layout.The input manager (542) may be configured to process user input (e.g., user input on a system screen or an app screen). The lightweight rendering engine (543) may be a renderer for generating images (e.g., the lightweight renderer (443)). For example, the lightweight rendering engine (543) may be used to display the system UI (530). In one embodiment, the spatialization manager (540) may include the lightweight rendering engine (543) for rendering the system UI. In one embodiment, when the lightweight rendering engine (543) does not have sufficient resources to render an avatar used in the HMD, at least one external rendering engine may be used. In this case, an external rendering engine (e.g., 3. rd To resolve compatibility issues with the party engine, an external rendering engine support module may be added within the spatialization manager (540).

[0119] According to one embodiment, the electronic device can execute an application. For example, in response to the execution of an XR application (510) (e.g., an XR application (442), a 3D game, an XR map, or other immersive application), the electronic device can execute a virtual space manager (550). The electronic device (101) can provide dual image information (561) provided from the XR application (510) to the virtual space manager (550). In order to display an image in a three-dimensional space, the dual image information (561) can include two pieces of image information that take binocular parallax into account. For example, the dual image information (561) can include first image information for the user's left eye and second image information for the user's right eye for rendering in a three-dimensional virtual space. Hereinafter, in the present disclosure, the term dual image information is used to refer to image information for displaying images for both eyes in a three-dimensional space. In addition to dual image information, the above dual image information may also include binocular image information, dual image information, dual image data, dual images, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimensional conversion image data, binocular parallax image data, and / or equivalent technical terms. The electronic device (101) can generate a composite image by merging image layers through a virtual space manager (550). The electronic device (101) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (250) of the electronic device (101).

[0120] According to one embodiment, the electronic device can execute at least one application among an XR application (510) and other applications (520) (e.g., a first application (520-1), a second application (520-2), ..., an Nth application (520-N)). According to one embodiment, the application (520) can be configured to output image information for displaying a two-dimensional image. In other words, the application (520) can provide a two-dimensional image. For example, the application (520) can be a video application, a schedule application, or an application (520) can be an Internet browser application. Let us assume that, in response to the execution of the application (520), image information (562) provided from the application (210) is provided to the virtual space manager (550). Since the image information (562) has only x-coordinates and y-coordinates within a two-dimensional plane, it may be difficult to consider the precedence relationship (i.e., the distance from the user) between other applications with respect to the user. The electronic device (101) may execute the spatialization manager (540) to provide dual image information to the virtual space manager (550) even when displaying an application (520) that provides a general 2D screen. For example, based on the execution of the spatialization manager (540), the electronic device (101) may receive application-related information (563) from the first application (520-1). For example, the application-related information (563) may include image information representing a 2D image of the first application (520-1) (e.g., information including RGB for each pixel) and / or content information in the first application (520-1) (e.g., characteristics of content executed in the first application, type of content). The application-related information (563) may be acquired through a spatialization API.Based on the execution of the spatialization manager (540), the electronic device (101) can identify information (hereinafter, location information) about the location of the area to be rendered and the size of the area to be rendered by the first application (520-1). Based on the execution of the spatialization manager (540), the electronic device (101) can generate dual image information (565, e.g., RGBx2) that takes into account the user's binocular disparity through the image information and the location information. Based on the execution of the spatialization manager (540), the electronic device (101) can provide the dual image information (565) to the virtual space manager (550). By converting a simple two-dimensional image into the dual image information (565), a problem that occurs when the image information (562) is directly transmitted to the virtual space manager (550) can be resolved. In addition, since at least some of the functions for displaying images in a virtual space are performed by the spatialization manager (540) instead of the virtual space manager (550), the burden on the virtual space manager (550) may be reduced. However, since the image information from the application (520) is not directly transmitted to the virtual space manager (550) but is transmitted through the spatialization manager (540), the quality of the image ultimately output to the user may be reduced. For example, in the first application (520-1), an image is rendered at a resolution of about 2756 x 1846, but the image may be downsampled in the process of being transmitted to the virtual space manager (550) through the spatialization manager (540) (e.g., downsampled from a resolution of about 2756 x 1846 to a resolution of about 1160 x 680). Thereafter, the virtual space manager (550) can upsample the downsampled image (e.g., upsample from a resolution of about 1160 x 680 to a resolution of about 1625 x 1070) and transfer the upsampled image to the display buffer.In this way, when an image is transferred from an application (520) to a spatialization manager (540) and then transferred from the spatialization manager (540) to a virtual space manager (550), a resolution mismatch may occur, or an aliasing problem or a deterioration in image quality may occur during the upsampling process. To solve the above-described problems, the present disclosure describes techniques for controlling the resolution of an area to be displayed in an application and performing foveation rendering based on the system structure illustrated in FIG. 5.

[0121] FIGS. 6A and 6B illustrate examples of block diagrams of electronic devices (e.g., electronic device (101), wearable device (101)) for controlling the resolution of an image in a virtual space. In FIGS. 6A and 6B, an example of executing multiple programs / instructions for controlling the resolution of an image in a virtual space is described. The multiple programs / instructions may all be executed in one processor (e.g., AP) or may be executed by multiple processors (e.g., AP, GPU (graphics processing unit), NPU (neural processing unit)). The meaning of being executed by multiple processors is that some programs / instructions may be executed by a first processor and other some programs / instructions may be executed by a second processor different from the first processor.

[0122] Referring to FIG. 6A, the electronic device (101) may execute a spatialization manager (540) and a virtual space manager (550) to render an image in a virtual space. For example, in response to the execution of an application (e.g., application (520)) configured to output a two-dimensional image, the electronic device (101) may request the spatialization manager (540) to provide the rendering size of the app to be executed, the type of content, and location information in order to generate dual image information (565) generated from the two-dimensional image (e.g., image information (563)) of the application. In other words, the electronic device (101) may be configured to convert the image information (563) of an application (e.g., application (520)) other than an immersive application (e.g., XR application (510)) into dual image information (565). The dual image information (565) may be generated based on the location information and image information (563) for the area to be rendered of the corresponding application. The electronic device (101) may execute a virtual space manager (550) to display the double image information in a three-dimensional virtual space. In the present disclosure, the location information indicates a location where an application is to be drawn in the virtual space (i.e., a location to be rendered), and may indicate xyz coordinates in the three-dimensional space. In addition to the location information, the location information may be referred to as rendering area information, location area information, location information, spatial information, coordinate information, area information, depth information, three-dimensional coordinate information, rendering information, distance information, z information, and / or terms having equivalent technical meanings thereto. For the spatialization manager (540) and the virtual space manager (550), the descriptions of FIG. 5 may be referred to.

[0123] The electronic device (101) may execute the resolution manager (630) to adaptively control the resolution depending on the location of the area where the application (520) is rendered, the size of the area, and / or the characteristics of the content being executed in the application (520). In the present disclosure, the characteristics of the content indicate the type, such as whether the content being executed in the application requires readability (e.g., text requires higher readability than a photograph) or whether the user's interest level is high (e.g., a face requires a higher interest level than an arm), and is hereinafter referred to as the type of content. The type of the content may be replaced with content information, characteristic information, object information, object type information, object characteristic information, sample information, image information, object category, object type, object characteristic, and / or equivalent technical terms. In response to the execution of the resolution manager (630), the electronic device (101) may determine the resolution for the application (520). For example, the electronic device (101) (e.g., spatialization manager (540)) may be requested, at the time when the application (520) is executed, information related to the application (520) to be executed (e.g., rendering size, type of content, location information), information related to foveation rendering (e.g., foveation area, foveation level, foveation method), and / or information about the user (e.g., user's area of ​​interest, gaze data, execution environment, whether pass-through is possible). The spatialization manager (540) may provide data (671) including the requested information to the resolution manager (630) (e.g., scaling, foveation manager) based on the request.

[0124] According to one embodiment, the resolution manager (630) may be configured to determine a resolution (e.g., final resolution (scaling factor), foveation level) for image rendering of the application (520) based on data. According to one embodiment, the resolution manager (630) may determine whether the type of content to be rendered in the application is a moving image and whether it includes text requiring readability. For example, if text requiring readability is included, the electronic device (101) may set the resolution relatively high. For example, if the content is a video or a photo, the electronic device (101) may set the resolution relatively low. The resolution manager (630) may provide resolution information (672) to the application (520). The resolution information (672) may indicate rendering settings guided to the application (520). The electronic device (101) may provide resolution information (e.g., resolution information (672)) indicating the resolution to the application (520). The above resolution information may include a scaling factor, a resolution factor, a resolution value, a resolution level, and / or equivalent technical parameters. The application (520) may change rendering settings according to the resolution information. The application (520) may change rendering settings according to the resolution information and provide an image according to the changed rendering settings. Hereinafter, a scaling factor is used as an example of resolution information in the present disclosure; however, in addition to the scaling factor, resolution factor, resolution adjustment parameter, resolution scaling factor, resolution control parameter, resolution parameter, and / or equivalent technical terms may be used instead.

[0125] According to one embodiment, the electronic device (101) (e.g., resolution manager (630)) may determine parameters for foveated rendering as well as determine a resolution for image rendering of the application (520). For example, the electronic device (101) may determine a foveation area and / or a foveation level based on the execution of the resolution manager (630). The foveation area may indicate an area to be displayed in high resolution in foveation rendering (e.g., may be referred to as a foveated area, a foveated portion, an area of ​​interest, a resolution concentration area, etc.). The foveation level may indicate the size of the foveated area relative to the entire area. Hereinafter, in the present disclosure, the foveation level may be replaced with a foveation resolution level, a resolution rendering level, a foveated rendering level, a foveated area ratio, a foveated area size, and / or equivalent technical terms in addition to the foveation level. According to one embodiment, the resolution manager (630) may determine the foveation area (foveation rendering) and / or the foveation level of the image to be finally displayed. According to one embodiment, the resolution manager (630) may provide a parameter (e.g., at least a part of the resolution information (672)) to the application (520). The electronic device (101) executing the application (520) may determine the rendering settings according to the provided information. The electronic device (101) may output the image layer to be finally rendered according to the rendering settings. The output image layer may be converted into dual image information through the spatialization manager (540). The converted dual image information may be rendered through the virtual space manager (550).

[0126] In one embodiment, the characteristics of content displayed on an application may change not only when the application is executed, but also while the application is executing. For example, when a user sees multiple applications displayed simultaneously, an operation may be performed to position the application of interest closest to the user. At this time, the resolution of the rendered image may be changed to increase the efficiency of GPU resources. For another example, applications such as Internet browsers may display various types of content. For example, the rendered content may change from a text page requiring readability to content such as a video or photo that does not require readability. Even in such cases, adaptive adjustment of the resolution of the displayed image may be required depending on the type of content. In one embodiment, an electronic device (e.g., spatialization manager (540)) may provide the changed content characteristics to the resolution manager (630). The resolution manager (630) may use the received information to change the rendering settings of the running application (520). The electronic device (101) (e.g., resolution manager (630)) can provide the rendering settings to the application (520). The spatialization manager (540) can convert the image information (563) of the application (520) according to the settings into dual image information (565). The spatialization manager (540) can also provide the dual image information (565) to the virtual space manager (550). As a non-limiting example, in a situation where all applications (e.g., XR application (510), application (520)) are running, when the foveation level is adjusted, the dual image information (651) of the XR application (520) can be provided to the virtual space manager (550) through the resolution manager (630).

[0127] The spatialization manager (540) may include a lightweight rendering engine (543) as a lightweight renderer (443) for rendering the system UI. However, the lightweight rendering engine (543) may have sufficient resources to render the system UI (530), but may not have sufficient resources to render a separate avatar. If an external rendering engine (e.g., an avatar rendering engine (660)) is used, there may be difficulties in displaying the avatar rendering result together with the system UI (530) due to incompatibility with the spatialization manager (540) structure.

[0128] For compatibility with external rendering engines, the spatialization manager (540) is rd A party support module (661) may be included. For example, since various rendering requirements such as overall creation, transformation, clothing change, and movement expression of an avatar to be rendered in the second application (520-2) are not satisfied, an external rendering engine (660) may be required. The electronic device (101) may include 3 of the spatialization manager (540). rdIn response to the execution of the party support module (661), the rendering results of an external rendering engine (e.g., an avatar rendering engine (660)) may be collected. Since the rendering results of the external rendering engine may not be compatible with the lightweight rendering engine (543), the rendering results may need to be converted into a format output by the lightweight rendering engine (543) or converted so that they can be rendered together. Through this process, the results of the external rendering engine can ultimately be expressed through the lightweight rendering engine (543). For example, avatar data and system UI-related data output from the lightweight rendering engine (543) can be collected through the spatialization manager (540). Based on the execution of the spatialization manager (540), the electronic device (101) can provide the collected data to the virtual space manager (550) in the format of dual image information (565) for rendering with binocular disparity.

[0129] Referring to FIG. 6B, the spatialization manager (540) may be configured to include a resolution manager (630). For example, the resolution manager (630) may be a component of the spatialization manager (540). According to one embodiment, the functions of the resolution manager (640) in FIG. 6A may be performed by the spatialization manager (540). For example, in a situation where all applications (e.g., XR application (510), application (520)) are running, when the fovea level is adjusted, the dual image information (565) of the XR application (520) may be transmitted to the virtual space manager (451) through the spatialization manager (540). Although not shown in FIG. 6b, when foveation rendering is not performed (e.g., when the foveation level is not set), the dual image information (565) of the XR application (520) can be directly transmitted to the virtual space manager (451) without passing through the spatialization manager (540).

[0130] Figure 7 shows examples of applications displayed in virtual space.

[0131] Referring to FIG. 7, according to one embodiment, the electronic device (101) can execute an application. For example, the electronic device (101) can execute a first application (710). The first application (710) may be an Internet browser. The electronic device (101) can execute a second application (720). The second application (720) may be a calendar application.

[0132] According to one embodiment, the electronic device (101) can display the applications being executed in the virtual space. The electronic device (101) can display a rendered image corresponding to image information provided by each application through the display. According to one embodiment, the electronic device (101) can determine the resolution of the image provided by the application according to the size and / or position of the area (hereinafter, “rendering area”) in the virtual space where the rendered image is to be displayed. The resolution does not indicate the resolution of the output image converted through image processing (e.g., downsampling or upsampling) in the spatialization manager (540) or the virtual space manager (550), but may refer to the resolution of the image provided at the application level. For example, the electronic device (101) can determine the resolution of the image of the first application (710). The electronic device (101) can determine the resolution according to the first depth (721), which is the distance from a reference point (e.g., a user's position) of the electronic device (101) to the first area. The electronic device (101) can convert the image information (e.g., a two-dimensional image) of the first application (710) corresponding to the determined resolution into dual image information (e.g., two two-dimensional images) for both eyes. Based on the dual image information, the electronic device (101) can display a rendered image in a virtual space through a display (e.g., the display (250)). The electronic device (101) can determine the resolution of the image of the second application (720). The electronic device (101) can determine the resolution according to the second depth (722), which is the distance from the reference point (e.g., the user's position) of the electronic device (101) to the second area. The electronic device (101) can convert the image information (e.g., a two-dimensional image) of the second application (720) corresponding to the determined resolution into dual image information (e.g., two two-dimensional images) for both eyes.The electronic device (101) may display a rendered image in a virtual space through a display (e.g., display (250)) based on the above-described dual image information. A second area in which a second application (720) is rendered in a three-dimensional virtual space may be located relatively farther away than a first area in which a first application (710) is rendered. Since the second depth (722) is deeper than the first depth (721), the electronic device (101) may set the resolution for the first application (710) higher than the resolution for the second application (720). This is because an activity closer to a user may be considered to have a higher level of interest from the user.

[0133] According to one embodiment, the resolution for an application may be determined based on the type of content provided by the application as well as the location of the rendering area. The resolution may not refer to the resolution of an output image converted through image processing (e.g., downsampling or upsampling) in the spatialization manager (540) or the virtual space manager (550), but may refer to the resolution of an image provided at the application level. For example, if the content of an application contains many characters that require reading, the electronic device (101) may set the resolution of the application relatively high. For example, the electronic device (101) may set the resolution for an e-book application higher than that for a video application. The electronic device (101) (e.g., the spatialization manager (540)) may obtain information about the type of content from an application (e.g., the application (520)) and determine a scaling factor according to the resolution of the application through the resolution manager (630). The electronic device (101) can request an image again from the application according to the determined scaling factor. The electronic device (101) can obtain an image (e.g., a two-dimensional image) generated according to the scaling factor from the application, and convert the generated image into dual image information having images for both eyes in a three-dimensional space. The electronic device (101) can display a rendered image based on the dual image information.

[0134] Although FIG. 7 illustrates an example in which the resolution of an image provided by an application is determined based on the location of the application or the characteristics of the content of the application, the embodiments of the present disclosure are not limited thereto. In addition to the above-described information, the electronic device (101) may also determine the resolution of an image provided by the application based on the gaze information of the user. For example, the electronic device (101) may obtain gaze data through at least one sensor (e.g., the gaze tracking camera (260-1) of FIGS. 2B and 3A, the image sensor (421) of FIG. 4). The electronic device (101) may set the resolution differently depending on how far the location indicated by the gaze data is from the location of the rendering area of ​​each application. For example, the electronic device (101) may set the resolution of the corresponding application lower as the direction toward the rendering area is farther away from the direction of the gaze data. As another example, if the location indicated by the user's gaze data is within a threshold range from the location of the application's rendering area, the electronic device (101) may determine the resolution of the image provided by the application based on the depth of the rendering area. If the location indicated by the user's gaze data is outside the threshold range from the location of the application's rendering area, the electronic device (101) may determine the resolution of the image provided by the application based on the depth of the rendering area according to a value set to a predetermined value. This is because, since it is outside the user's area of ​​interest, not adjusting the resolution may be efficient in terms of resource management.

[0135] Figure 8 shows examples of images for foveated rendering. Foveated rendering refers to a technique of dividing an image into a foveated area and a residual area, synthesizing a high-resolution image corresponding to the foveated area and a low-resolution image corresponding to the residual area, and rendering the synthesized image.

[0136] Referring to FIG. 8, the electronic device (101) can execute an application (e.g., application (520)). The application may be an application configured to provide a two-dimensional image. For example, the two-dimensional image provided through the application may be an image (810).

[0137] According to one embodiment, the electronic device (101) can perform foveated rendering on the image (810). The electronic device (101) can determine parameters for the foveated rendering. According to one embodiment, the electronic device (101) can determine the size of the foveated area (817) for the foveated rendering. For example, the electronic device (101) can determine the size of the foveated area (817) according to the location of the rendering area of ​​the application. For example, since the resolution decreases as the rendering area is located farther away, the size of the foveated area (817) can be set to be relatively large in order to increase the user's visibility. For example, the electronic device (101) can determine the size of the foveated area (817) according to the size of the rendering area of ​​the application. For example, as the size of the rendering area increases, the size of the foveated area (817) can also be set to be large. As another example, even if the size of the rendering area increases, the size of the foveated area (817) may be fixed. In other words, as the size of the rendering area increases, the foveation level may decrease. Here, the foveation level may represent the size of the foveated area (817) relative to the overall image size. For example, the electronic device (101) may determine the size of the foveated area (817) according to the type of content of the application. For example, if the content includes a human face, the electronic device (101) may set the area including the face as the region of interest. As another example, if the content includes letters, the electronic device (101) may set the area including the letters as the region of interest. If a user input for scrolling a page or a user input for enlarging a specific area is received, the electronic device (101) may also change the size of the region of interest.At this time, the electronic device (101) can determine the size of the foveated area (817) corresponding to the size of the region of interest. For example, the electronic device (101) can determine the size of the foveated area (817) according to the user's gaze data. The electronic device (101) can set the size of the foveated area (817) to be larger as the position of the user's gaze data is closer to the position of the application's rendering area.

[0138] According to one embodiment, the electronic device (101) can determine the location of the foveated area (817) for the foveated rendering. For example, the electronic device (101) can determine the location of the foveated area (817) according to the type of content. For example, if the content includes a human face, the electronic device (101) can set the location of the face as the region of interest. As another example, if the content includes letters, the electronic device (101) can set the region including the letters as the region of interest. If a user input for scrolling a page or a user input for enlarging a specific region is received, the electronic device (101) can also change the location of the region of interest. The electronic device (101) can determine the region of interest as the foveated area (817). For example, the electronic device (101) can determine the location of the foveated area (817) according to the user's gaze data. The electronic device (101) can set an area within the entire image, within the position of the user's gaze data and a critical distance, as a foveated area (817).

[0139] According to one embodiment, after parameter(s) for foveated rendering are determined, the electronic device (101) can provide information about the parameter(s) to the application. Through the application, the electronic device (101) can generate a composite image for the foveated rendering. Based on the execution of the application, the electronic device (101) can obtain a high-resolution first image (820) for the foveated area (817). Based on the execution of the application, the electronic device (101) can obtain a low-resolution second image (830) for the remaining area. Based on the execution of the application, the electronic device (101) can obtain a composite image by combining the first image (820) and the second image (830). The electronic device (101) can convert the composite image (e.g., a two-dimensional image) into dual image information (e.g., two two-dimensional images). For example, the electronic device (101) can convert image information corresponding to the composite image into dual image information corresponding to images for both eyes through the spatialization manager (540) of FIGS. 6A and 6B. The electronic device (101) can display rendered images corresponding to the dual image information through a display (e.g., display (250)).

[0140] Figure 9 shows an example of resolution change according to movement of an application in virtual space.

[0141] Referring to FIG. 9, according to one embodiment, an electronic device (101) may execute an application. For example, the application may be an Internet browser. The electronic device (101) may display the executed application in a virtual space. The electronic device (101) may display a rendered image corresponding to image information provided by the application through a display. According to one embodiment, the electronic device (101) may determine the resolution of the image provided by the application based on the size and / or position of an area (i.e., a rendering area) in the virtual space where the rendered image is to be displayed. The resolution may not indicate the resolution of an output image converted through image processing (e.g., downsampling or upsampling) in the spatialization manager (540) or the virtual space manager (550), but may refer to the resolution of an image provided at the application level.

[0142] An electronic device (101) may receive a user input for moving a rendering area of ​​an application located at a first location (910) to a second location (920). Prior to receiving the user input, the resolution of an image of the application may be a first value. For example, the first value may be determined based on a first depth (921), which is a distance from a reference point of the electronic device (101) (e.g., a user's location) to the first location.

[0143] An electronic device (101) (e.g., spatialization manager (540)) can detect a change in the location of an application. As the location of the application changes, the application can be configured to inquire the spatialization manager (540) about the display quality according to the changed location. The electronic device (101) (e.g., spatialization manager (540)) can obtain location information of the application in response to the user input. For example, the electronic device (101) can obtain information about a second depth (922), which is a distance from a reference point of the electronic device (101) (e.g., a user's location) to a second location. As a non-limiting example, when the resolution manager (630) is located outside the spatialization manager (540), the spatialization manager (540) can provide information about the second depth (922) to the resolution manager (630). The electronic device (101) (e.g., spatialization manager (540), resolution manager (630)) can change the resolution according to the second depth (922).

[0144] The electronic device (101) (e.g., spatialization manager (540), resolution manager (630)) may provide resolution information (e.g., resolution information (672)) to the application in response to an inquiry from the application. The electronic device (101) may provide resolution information (e.g., resolution information (672)) indicating the changed resolution to the application.

[0145] The electronic device (101) can generate a two-dimensional image of the application according to the changed resolution of the resolution information through the running application. Even if the rendering area of ​​the application moves, the electronic device (101) can obtain a two-dimensional image according to the resolution determined at the application level by re-performing the processing procedures in the order of the spatialization manager (540), the resolution manager (630), and the application (520) of FIGS. 6A and 6B. The electronic device (101) can convert image information corresponding to the generated two-dimensional image into dual image information (e.g., two two-dimensional images) for both eyes. Based on the dual image information, the electronic device (101) can display a rendering image in a virtual space through a display (e.g., the display (250)).

[0146] Although FIG. 9 illustrates an example of changing the rendering quality by moving the rendering area of ​​an application according to a user input, the embodiments of the present disclosure are not limited thereto. It may also be understood that not only the case where the rendering area of ​​an application moves, but also the case where the rendering quality of an application changes as the distance between the user and the rendering area of ​​the application changes as the user moves may be an embodiment of the present disclosure. For example, in response to the movement of the user, the electronic device (101) may detect a change in depth information of the rendering area of ​​the application. The application may inquire about the display quality according to the change in the depth information by requesting the spatialization manager (540) of the electronic device (101) for information related to the display quality (e.g., resolution information (672) indicating a changed resolution). The spatialization manager (540) (or resolution manager (630)) may provide the application with information related to the display quality (e.g., resolution information (672) indicating a changed resolution). The electronic device (101) can be configured to output a rendering image according to the display quality through the application.

[0147] Figure 10 shows the operation flow of an electronic device (e.g., electronic device (101), wearable device (101)) for controlling the resolution of an application in a virtual space.

[0148] Referring to FIG. 10, in operation (1001), the electronic device (101) (e.g., spatialization manager (540)) may acquire depth information about an area (hereinafter, rendering area) in which the application is to be rendered in a three-dimensional space in response to the execution of an application (e.g., an Internet browser, a gallery, a calendar, a settings application, a two-dimensional map) configured to provide a two-dimensional image. The electronic device (101) (e.g., spatialization manager (540)) may acquire depth information to control the resolution of a two-dimensional image in the three-dimensional space. Here, the depth information may represent a distance (e.g., depth, Z-coordinate, Z-depth) between a user of the electronic device (101) and the rendering area in the three-dimensional virtual space. As a non-limiting example, if the position of the rendering area changes according to the execution of another application or a user's input, the electronic device (101) may acquire depth information again.

[0149] In operation (1003), the electronic device (101) (e.g., resolution manager (630)) may determine resolution information for the application based on depth information. The resolution for the application may not refer to the resolution of an output image converted through separate image processing (e.g., downsampling of the spatialization manager (540) of FIG. 5 or upsampling of the virtual space manager (550)), but may refer to the resolution of an image provided from the application at the application level.

[0150] The electronic device (101) (e.g., resolution manager (630)) can determine resolution information for an application based on depth information of the application. For example, the electronic device (101) can set a higher resolution for the application as the rendering area of ​​the application gets closer to the user. The method for setting the resolution according to the location of the rendering area can be configured in various ways. For example, the electronic device (101) can determine the resolution of the application through a function set to be inversely proportional to the depth of the rendering area. As another example, the electronic device (101) can pre-store a plurality of resolution levels according to a plurality of distance ranges. The electronic device (101) can identify a distance range corresponding to the depth of the rendering area among the plurality of distance ranges. The electronic device (101) can identify a resolution level corresponding to the distance range. As another example, the electronic device 9101) may determine the resolution for the application through conditions, functions, or learning models that take into account not only the location of the rendering area, but also the size of the rendering area, information about the type of content of the application, user input, the number of other running application(s), the type of other running applications, and / or the user's gaze data.

[0151] In operation (1005), the electronic device (101) (e.g., the spatialization manager (540)) can convert image information corresponding to a two-dimensional image generated according to resolution information into dual image information corresponding to images for both eyes. The electronic device (101) can generate a two-dimensional image based on the resolution information. The electronic device (101) (e.g., the spatialization manager (540)) can generate image information having a two-dimensional image according to the resolution information through the application. The electronic device (101) (e.g., the spatialization manager (540)) can convert the image information into dual image information. The dual image information can include first image information for the left eye (e.g., information about a two-dimensional image to be displayed on the left eye) and second image information for the right eye (e.g., information about a two-dimensional image to be displayed on the right eye).

[0152] In operation (1007), the electronic device (101) can display a rendered image based on the dual image information. The electronic device (101) (e.g., the virtual space manager (550)) can obtain the dual image information converted by the spatialization manager (540). The electronic device (101) (e.g., the virtual space manager (550)) can provide the rendered images corresponding to the dual image information to the display buffer. For example, the electronic device (101) can display another application, an avatar, and / or a system UI (e.g., the system UI (530)) in the virtual space simultaneously with the image for the application. The electronic device (101) can obtain the rendered images by synthesizing the image layers corresponding to the dual image information with layers for another application, an avatar, and / or a system UI (e.g., the system UI (530)). The electronic device (101) can display the above-described rendering images through a display (e.g., a first display (250-1), a second display (250-2)).

[0153] Through FIGS. 5 to 10, embodiments have been described in which the rendering quality for an application is determined based on depth information of the area in which the application is to be rendered. However, embodiments of the present disclosure are not limited thereto. According to one embodiment, a rendering area that should be maintained regardless of the distance between applications may be set. For example, in the case of advertisements or warning messages, a constant rendering quality (e.g., maintaining a resolution above a certain value) may be required regardless of the distance between the user and the application. The electronic device (101) (e.g., spatialization manager (540)) may be configured not to change the resolution or rendering settings even if the depth information of the application changes based on information about the content type of the application and / or the type of the application. In other words, the electronic device (101) may set an exception processing for quality change based on depth information for a specific type of content or a designated type of application.

[0154] In the present disclosure, a technique for displaying an image without a breakup phenomenon due to resolution while maintaining readability is described even when an application configured to display a two-dimensional image is executed. By controlling the resolution according to the distance of the rendering area and controlling the foveated level for foveated rendering, the electronic device (101) can adjust the quality of the content to be displayed on the built-in display and provide the user with more visible content. In addition, a separate support module (e.g., 3 rd Through the party support module (661), in the spatialization manager (540)-virtual space manager (550) structure, avatar rendering and system UI (530) can be made to operate smoothly and simultaneously. By appropriately setting the resolution and foveated level, optimization can be achieved between the use of resources (e.g., GPU resources) that are in a trade-off relationship and user visibility.

[0155] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0156] In embodiments, an electronic device is provided. The electronic device may include at least one display, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device, in response to execution of an application configured to provide a two-dimensional image, to obtain depth information for an area in a three-dimensional space where the application is to be rendered, determine resolution information for the application based on the depth information for the area, convert image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes, and display a rendered image through the at least one display based on the dual image information.

[0157] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive a user input for changing a location of an area in which the application is to be rendered in the three-dimensional space from a first location to a second location, in response to the user input, obtain information about the second location, change a resolution according to the information about the second location, convert second image information corresponding to the two-dimensional image generated according to resolution information representing the changed resolution into second dual image information corresponding to images for both eyes, and display a second rendered image through the at least one display based on the second dual image information.

[0158] For example, the depth information for the area where the application is to be rendered may indicate the depth from a reference point to the area where the application is to be rendered in the three-dimensional space. The resolution according to the resolution information may be determined to be lower as the depth from the reference point to the area where the application is to be rendered increases.

[0159] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to, in response to execution of a second application configured to provide a two-dimensional image, acquire second depth information for a second area in the three-dimensional space where the second application is to be executed, determine second resolution information representing a resolution for the second application based on the second depth information for the second area, generate second image information corresponding to the two-dimensional image of the second application generated based on the second resolution information, and generate second dual image information corresponding to images for both eyes, and display a second rendered image through the at least one display based on the second dual image information. When a depth represented by the depth information is greater than a depth represented by the second depth information, a resolution of the second application based on the second resolution information may be higher than a resolution of the application based on the resolution information.

[0160] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device, in response to the execution of the application, to identify a type of content of the application and, based on information about the type of content of the application and depth information for the region, determine resolution information for the application. The information about the type of content of the application may indicate whether a region of the content includes characters.

[0161] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to change a resolution representing the resolution based on information about the type of the changed content, based on identifying that content running in the application has changed, generate corrected image information corresponding to a two-dimensional image of the application according to the resolution information representing the changed resolution, convert the corrected image information into corrected dual image information corresponding to images for both eyes, and display a rendered image based on the corrected dual image information.

[0162] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a foveation level for foveated rendering of the two-dimensional image based on depth information for the area, generate image information corresponding to the two-dimensional image of the application based on the resolution information and the foveation level, and convert the generated image information into dual image information corresponding to images for the two eyes. The foveation level may represent a size of the foveated area for a high-resolution foveated area and a low-resolution remaining area of ​​the two-dimensional image.

[0163] For example, the foveation level may be determined based on depth information for the area, information about the type of content in the application, and the user's gaze data. The resolution information may indicate the resolution of the foveated area and the resolution of the remaining areas. The user's gaze data may be acquired through an image sensor of the electronic device.

[0164] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a first value as the foveation level if the location of the area is within a threshold range from a location according to the user's gaze data, and to determine a second value as the foveation level if the location of the area is outside the threshold range from the location according to the user's gaze data. A size of the foveated area at the foveation level corresponding to the first value may be greater than a size of the foveated area at the foveation level corresponding to the second value.

[0165] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine the resolution information according to a depth indicated by the depth information when the location of the area is within a threshold range from a location according to the user's gaze data, and to determine the resolution information as a predetermined value when the location of the area is outside the threshold range from a location according to the user's gaze data.

[0166] For example, the application may be configured to provide the image information by changing the rendering quality of the two-dimensional image based on the resolution information. The dual image information may be converted from the provided image information according to the changed rendering quality.

[0167] For example, the above resolution information may include resolution information.

[0168] In embodiments, a method performed by an electronic device is provided. The method may include, in response to execution of an application configured to provide a two-dimensional image, acquiring depth information for an area in a three-dimensional space where the application is to be rendered; determining resolution information for the application based on the depth information for the area; converting image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes; and displaying a rendered image based on the dual image information.

[0169] For example, the method may include an operation of receiving a user input for changing a location of an area in which the application is to be rendered in the three-dimensional space from a first location to a second location, an operation of obtaining information about the second location in response to the user input, an operation of changing a resolution according to the information about the second location, an operation of converting second image information corresponding to the two-dimensional image generated according to resolution information indicating the changed resolution into second dual image information corresponding to images for both eyes, and an operation of displaying a rendered image based on the second dual image information.

[0170] For example, the depth information for the area where the application is to be rendered may indicate the depth from a reference point to the area where the application is to be rendered in the three-dimensional space. The resolution according to the resolution information may be determined to be lower as the depth from the reference point to the area where the application is to be rendered increases.

[0171] For example, the method may include, in response to execution of a second application configured to provide a two-dimensional image, acquiring second depth information for a second area in the three-dimensional space where the second application is to be executed; determining second resolution information representing a resolution for the second application based on the second depth information for the second area; generating second image information corresponding to the two-dimensional image of the second application generated based on the second resolution information, second dual image information corresponding to images for both eyes; and displaying a second rendered image based on the second dual image information. When a depth represented by the depth information is greater than a depth represented by the second depth information, a resolution of the second application based on the second resolution information may be higher than a resolution of the application based on the resolution information.

[0172] For example, the operation of determining the resolution information may include, in response to the execution of the application, identifying the type of content of the application, and determining resolution information for the application based on information about the type of content of the application and depth information for the region. Information about the type of content of the application may indicate whether a region of the content includes characters.

[0173] For example, the method may include an operation of changing a resolution representing the resolution based on information about the type of the changed content, based on identifying that the content running in the application has changed, an operation of generating corrected image information corresponding to a two-dimensional image of the application according to the resolution information representing the changed resolution, an operation of converting the corrected image information into corrected dual image information corresponding to images for both eyes, and an operation of displaying a rendered image based on the corrected dual image information.

[0174] For example, the operation of generating the dual image information may include an operation of determining a foveation level for foveated rendering of the two-dimensional image based on depth information for the area, an operation of generating the image information corresponding to the two-dimensional image of the application based on the resolution information and the foveation level, and an operation of converting the generated image information into dual image information for displaying images for the two eyes. The foveation level may indicate the size of the foveated area for a high-resolution foveated area and a low-resolution remaining area of ​​the two-dimensional image.

[0175] For example, the foveation level may be determined based on depth information for the area, information about the type of content in the application, and the user's gaze data. The resolution information may indicate the resolution of the foveated area and the resolution of the remaining areas. The user's gaze data may be acquired through an image sensor of the electronic device.

[0176] For example, the operation of determining the foveation level may include an operation of determining a first value as the foveation level when the location of the area is within a threshold range from a location according to the user's gaze data, and an operation of determining a second value as the foveation level when the location of the area is outside the threshold range from the location according to the user's gaze data. The size of the foveated area at the foveation level corresponding to the first value may be larger than the size of the foveated area at the foveation level corresponding to the second value.

[0177] In embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include a memory configured to store instructions, the memory including one or more storage media. The instructions, when individually or collectively executed by at least one processor, may cause an electronic device to, in response to execution of an application configured to provide a two-dimensional image, obtain depth information for an area in three-dimensional space where the application is to be rendered, determine resolution information for the application based on the depth information for the area, convert image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes, and display a rendered image based on the dual image information.

[0178] In embodiments, an electronic device is provided. The electronic device may include at least one display and at least one processor including a processing circuit. The at least one processor may be configured to, in response to execution of an application configured to provide a two-dimensional image, obtain depth information for an area in a three-dimensional space where the application is to be rendered, determine resolution information for the application based on the depth information for the area, convert image information corresponding to the two-dimensional image of the application generated based on the resolution information into dual image information corresponding to images for both eyes, and display a rendered image through the at least one display based on the dual image information.

[0179] In embodiments, an electronic device is provided. The electronic device may include a spatialization manager for obtaining spatial information about an application; a resolution manager for determining a resolution for the application; and a virtual space manager for providing an image for the application in a virtual space to a display buffer. The spatialization manager may be configured to obtain depth information about an area in which the application is to be rendered in a three-dimensional space. The resolution manager may be configured to determine resolution information for the application based on the depth information about the area in which the application is to be rendered, and provide the determined resolution information for the application to the application. The spatialization manager may be configured to convert image information generated by the application based on the resolution information into dual image information corresponding to images for both eyes. The virtual space manager may be configured to provide rendered images based on the dual image information to the display buffer.

[0180] For one or more embodiments, at least one of the components described in one or more of the preceding drawings may be configured to perform one or more operations, techniques, processes, and / or methods as described herein. For example, a processor (e.g., a baseband processor) described herein with respect to one or more of the preceding drawings may be configured to operate according to one or more examples described herein. For another example, circuitry associated with a user equipment (UE), a base station, a network element, and the like, as described above with respect to one or more of the preceding drawings, may be configured to operate according to one or more examples described herein.

[0181] Any of the embodiments described above may be combined with any other embodiment (or combination of embodiments) unless explicitly stated otherwise. The foregoing description of one or more implementations provides examples and descriptions, but is not intended to be exhaustive or limit the scope of the embodiments to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be learned from practicing various embodiments.

[0182] For one or more embodiments, at least one of the components described in one or more of the preceding drawings may be configured to perform one or more operations, techniques, processes, and / or methods as described herein. For example, a processor (e.g., a baseband processor) described herein with respect to one or more of the preceding drawings may be configured to operate according to one or more examples described herein. For another example, circuitry associated with a user equipment (UE), a base station, a network element, and the like, as described above with respect to one or more of the preceding drawings, may be configured to operate according to one or more examples described herein.

[0183] Any of the embodiments described above may be combined with any other embodiment (or combination of embodiments) unless explicitly stated otherwise. The foregoing description of one or more implementations provides examples and descriptions, but is not intended to be exhaustive or limit the scope of the embodiments to the precise forms disclosed. Modifications and variations are possible in light of the above teachings or may be learned from practicing various embodiments.

[0184] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0185] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0186] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0187] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0188] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0189] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, At least one display; At least one processor comprising a processing circuit; and A memory comprising one or more storage media for storing instructions, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: In response to execution of an application configured to provide a two-dimensional image, acquiring depth information about an area in three-dimensional space where the application is to be rendered; Based on the depth information for the above area, the resolution information for the above application is determined, Converting image information corresponding to the two-dimensional image of the application generated according to the above resolution information into dual image information corresponding to images for both eyes, Causing a rendering image to be displayed through at least one display based on the above dual image information; Electronic devices.

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Receiving a user input for changing the location of the area where the application is to be rendered in the three-dimensional space from a first location to a second location, In response to the above user input, information about the second location is obtained, Depending on the information about the second location above, change the resolution, Converting second image information corresponding to the two-dimensional image generated according to the resolution information indicating the changed resolution into second dual image information corresponding to images for both eyes, Causing a second rendering image to be displayed through at least one display based on the second dual image information; Electronic devices.

3. In claim 1, Depth information for the area where the above application is to be rendered represents the depth from the reference point to the area where the above application is to be rendered in the three-dimensional space. The resolution according to the above resolution information is determined to be lower as the depth from the reference point to the area where the application is to be rendered increases. Electronic devices.

4. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: In response to execution of a second application configured to provide a two-dimensional image, second depth information is acquired for a second area in which the second application is to be executed in the three-dimensional space; According to the second depth information for the second area, second resolution information indicating the resolution for the second application is determined, Generate second image information corresponding to the two-dimensional image of the second application, which is generated according to the second resolution information, and generate second dual image information corresponding to images for both eyes, Causing a second rendering image to be displayed through at least one display based on the second dual image information; If the depth indicated by the above depth information is greater than the depth indicated by the second depth information, the resolution of the second application according to the second resolution information is higher than the resolution of the application according to the resolution information. Electronic devices.

5. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: In response to the execution of said application, identifying the type of content of said application, Causing resolution information representing a resolution for the application to be determined based on information about the type of content of the application and depth information about the area; Information about the type of the content of the above application, indicating whether an area of ​​the content contains characters, Electronic devices.

6. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on identifying that the content running in the above application has changed, and based on information about the type of the changed content, changing the resolution, Generate modified image information corresponding to the two-dimensional image of the application according to the resolution information indicating the changed resolution, Convert the above modified image information into modified dual image information corresponding to images for both eyes, Causing the rendering image to be displayed based on the above modified double image information, Electronic devices.

7. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on the depth information for the above region, a foveation level for foveated rendering of the two-dimensional image is determined, Based on the above resolution information and the foveation level, the image information corresponding to the two-dimensional image of the application is generated, Causing the generated image information to be converted into dual image information corresponding to the images for the two eyes, The above foveation level represents the size of the foveated area relative to the high-resolution foveated area and the remaining low-resolution area of ​​the two-dimensional image. Electronic devices.

8. In claim 7, The above foveation level is determined based on depth information for the area, information about the type of content on the application, and the user's gaze data. The above resolution information represents the resolution of the foveated area and the resolution of the remaining area, The above user's gaze data is obtained through the image sensor of the electronic device. Electronic devices.

9. In claim 7, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: If the location of the above area is within a critical range from the location according to the user's gaze data, a first value is determined as the foveation level, If the location of the above area is outside the critical range from the location according to the user's gaze data, it causes a second value to be determined as the foveation level, The size of the foveated area at the foveation level corresponding to the first value is larger than the size of the foveated area at the foveation level corresponding to the second value. Electronic devices.

10. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: If the location of the above area is within a critical range from the location according to the user's gaze data, the resolution information is determined according to the depth indicated by the depth information, If the location of the above area is outside the critical range from the location according to the user's gaze data, causing the resolution information to be determined as a predetermined value. Electronic devices.

11. In claim 1, The above application is configured to provide the image information by changing the rendering quality of the two-dimensional image based on the resolution information, The above double image information is converted from the provided image information according to the changed rendering quality. Electronic devices.

12. In claim 1, The above resolution information includes a scaling factor, Electronic devices.

13. In a method performed by an electronic device, In response to the execution of an application configured to provide a two-dimensional image, an operation of obtaining depth information about an area in which the application is to be rendered in three-dimensional space; An operation for determining resolution information for the application based on depth information for the above area, An operation of converting image information corresponding to the two-dimensional image of the application generated according to the above resolution information into dual image information corresponding to images for both eyes, Including an action of displaying a rendered image based on the above dual image information, method.

14. In claim 13, An action of receiving a user input for changing the location of an area in which the application is to be rendered from a first location to a second location in the three-dimensional space, In response to the above user input, an operation of obtaining information about the second location; According to the information about the second location above, an action to change the resolution, An operation of converting second image information corresponding to the two-dimensional image generated according to resolution information indicating the changed resolution into second dual image information corresponding to images for both eyes; Further comprising an action of displaying a rendered image based on the second dual image information. method.

15. In claim 13, Depth information for the area where the above application is to be rendered represents the depth from the reference point in the three-dimensional space to the area where the above application is to be rendered. The resolution according to the above resolution information is determined to be lower as the depth from the reference point to the area where the application is to be rendered increases. method.

Images