Wearable device and method for rendering image, and non-transitory computer-readable storage medium

Abstract

This wearable device may comprise: a memory for storing instructions; at least one first camera; at least one second camera; a display assembly that includes at least one display including a display area; and at least one processor. The at least one processor can cause the wearable device to: identify an event for displaying a screen; on the basis of the event for displaying the screen within at least one portion of the display area where a user is gazing, display the screen within the at least one portion of the display area according to rendering of an image by using depth values; and, on the basis of the event for displaying the screen outside the at least one portion of the display area where the user is gazing, display the screen outside the at least one portion of the display area according to rendering of the image by using some of the depth values.

Description

Wearable device, method, and non-transitory computer-readable storage medium for rendering images

[0001] The present disclosure relates to a wearable device, a method, and a non-transitory computer-readable storage medium for rendering an image.

[0002] A wearable device may include a camera and a display. The wearable device may be utilized as a tool for implementing virtual reality, augmented reality, and mixed reality. The wearable device may display an image captured by the camera on the display. The wearable device may provide a clear image by performing correction on the image captured by the camera.

[0003] The above information may be provided as background art to aid in understanding the present disclosure.

[0004] No claim or determination is made as to whether any of the above is applicable as prior art to the present disclosure.

[0005] A wearable device is described. The wearable device may include a memory storing instructions and including one or more storage media. The wearable device may include at least one first camera. The wearable device may include at least one second camera. The wearable device may include a display assembly including at least one display including a display area. The wearable device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify an event that displays a screen generated using an image acquired through the at least one first camera. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen within at least a portion of the display area, based on the event of displaying the screen within at least a portion of the display area where the gaze of the user identified through the at least one second camera is located, by rendering the image using depth values ​​acquired in connection with acquiring the image. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen outside the at least portion of the display area, based on the event of displaying the screen outside the at least portion of the display area where the gaze of the user identified through the at least one second camera is located, by rendering the image using a portion of the depth values.

[0006] A method is provided. The method can be executed in a wearable device having a display assembly including at least one first camera, at least one second camera, and at least one display including a display area. The method can include an operation of identifying an event of displaying a screen generated using an image acquired through the at least one first camera. The method can include an operation of displaying the screen within at least a portion of the display area, based on the event of displaying the screen within at least a portion of the display area where the gaze of a user identified through the at least one second camera is located, by rendering the image using depth values ​​acquired in connection with acquiring the image. The method can include an operation of displaying the screen outside at least a portion of the display area, based on the event of displaying the screen outside at least a portion of the display area where the gaze of a user identified through the at least one second camera is located, by rendering the image using a portion of the depth values.

[0007] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device having a display assembly including at least one first camera, at least one second camera, and at least one display including a display area, cause the wearable device to identify an event of displaying a screen generated using an image acquired through the at least one first camera. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the screen within at least a portion of the display area based on the event of displaying the screen within at least a portion of the display area where a user's gaze identified through the at least one second camera is located, by rendering the image using depth values ​​acquired in connection with acquiring the image. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the screen outside the at least part of the display area by rendering the image using a portion of the depth values ​​based on the event of displaying the screen outside the at least part of the display area where the gaze identified through the at least one second camera is located.

[0008] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0009] FIG. 1 is a diagram illustrating an example of an environment including a wearable device according to various embodiments;

[0010] FIG. 2 is a diagram illustrating an example of a wearable device displaying a screen according to various embodiments;

[0011] FIG. 3 is a block diagram illustrating an exemplary configuration of an exemplary wearable device according to various embodiments;

[0012] FIG. 4 is a flowchart illustrating an exemplary method of displaying a screen according to various embodiments;

[0013] FIGS. 5A and 5B are drawings illustrating examples of rendering images according to various embodiments;

[0014] FIG. 6A is a diagram illustrating an example explaining the characteristics of each mode according to various embodiments;

[0015] FIG. 6b is a diagram illustrating examples of events related to recommended areas according to various embodiments;

[0016] FIG. 7 is a diagram illustrating examples of events related to content according to various embodiments;

[0017] FIG. 8 is a diagram illustrating an example of controlling the size of an area related to a foveated rendering mode according to various embodiments;

[0018] FIG. 9 is a block diagram illustrating an exemplary electronic device within a network environment according to various embodiments;

[0019] FIG. 10A is a perspective view illustrating an exemplary wearable device according to various embodiments;

[0020] FIG. 10b is a perspective view illustrating an example of one or more hardware elements arranged within a wearable device according to various embodiments;

[0021] FIGS. 11A and 11B are perspective views illustrating examples of the appearance of a wearable device according to various embodiments;

[0022] FIG. 12 is a block diagram illustrating an exemplary configuration of a wearable device according to various embodiments; and

[0023] FIG. 13 is a block diagram illustrating an exemplary configuration of an electronic device for displaying an image in a virtual space according to various embodiments.

[0024] FIG. 1 is a diagram illustrating an example of an environment including a wearable device according to various embodiments.

[0025] Referring to FIG. 1, a wearable device (100) may be used to provide a three-dimensional (3D) environment. For example, the wearable device (100) may be worn by a user (120). For example, the wearable device (100) may be worn on the head of the user (120). For example, the wearable device (100) may be referred to as a head-wearable electronic device. For example, the wearable device (100) may include a display assembly (e.g., the display assembly (308) of FIG. 3). For example, the wearable device (100) may provide an image corresponding to an external environment through the display assembly. For example, the wearable device (100) may provide an image acquired through at least one first camera (e.g., at least one first camera (309) of FIG. 3) through the display assembly.

[0026] According to one embodiment, the environment (130) may include a user (120), a wearable device (100), a sofa (140), a table (150), an air conditioner (160), a refrigerator (180), and / or a window (170). For example, the wearable device (100) may acquire an image of the environment (130) through the at least one first camera. For example, the wearable device (100) may provide the image acquired through the at least one first camera through the display assembly. For example, the wearable device (100) may provide or execute a pass-through function. For example, the pass-through function may be described as a function of providing a visual object corresponding to the external environment of the wearable device (100) through the display assembly. For example, the pass-through function may be described as a function of providing an image acquired through at least one first camera (e.g., at least one first camera (309) of FIG. 3) of the wearable device (100) through a display assembly (e.g., the display assembly (308) of FIG. 3). For example, the wearable device (100) may, based on executing the pass-through function, display images of the environment (130) acquired through at least one first camera (e.g., at least one first camera (309) of FIG. 3) in real time through a display assembly (e.g., the display assembly (308) of FIG. 3), thereby providing a user experience similar to viewing the environment (130) with the user's (120's) eyes while wearing the wearable device (100).

[0027] The scene of the environment (130) seen by the user (120) may differ from the images of the environment (130) provided when the wearable device (100) executes the pass-through function. For example, the user (120) may feel dizzy as the difference between the scene and the images increases. For example, to prevent and / or reduce the dizziness, the wearable device (100) may be required to provide images similar to the scene. For example, the wearable device (100) may perform image processing techniques to provide images similar to the scene. For example, the image processing techniques may include resolution enhancement, noise removal, brightness adjustment, and color tuning. For example, the image processing techniques may include minimizing and / or reducing spatial distortion. For example, the wearable device (100) may apply depth values ​​(e.g., depth values ​​(550) of FIG. 5B) to each of the objects included in the images (e.g., visual objects (240) to (280) of FIG. 5B) in order to minimize and / or reduce the spatial distortion. For example, a technique for minimizing and / or reducing the spatial distortion may be described as performing interpolation on the edges of each of the objects. For example, as the interpolation is performed, the spatial distortion may occur. For example, the user (120) may perceive dizziness due to the spatial distortion. For example, the wearable device (100) may provide the user (120) with images similar to the scene by applying the depth values ​​to the objects.

[0028] According to one embodiment, the wearable device (100) may provide or display an image of the external environment through the display assembly. For example, the display assembly may include at least one opaque display. For example, since the display assembly includes at least one opaque display, the wearable device (100) may be required to provide an image of the environment (130) through the display assembly. For example, while the wearable device (100) provides an image of the environment (130) through the display assembly, the wearable device (100) may provide content (e.g., content (220) of FIG. 2 ). The provision of the image and content is described and exemplified in more detail with reference to FIG. 2 .

[0029] FIG. 2 is a diagram illustrating an example of a wearable device displaying a screen according to various embodiments.

[0030] Referring to FIG. 2, the wearable device (100) may provide content (220) through a display assembly (e.g., display assembly (308) of FIG. 3) while providing a pass-through function. For example, state (210) may be described as a state in which content (220) is displayed while the pass-through function is being executed through the display assembly. For example, the wearable device (100) may provide content (220) while providing an image or screen corresponding to an environment (130) through the display assembly. For example, the wearable device (100) may provide content (220) as an overlay on an image or screen corresponding to the environment (130). However, the disclosure is not limited thereto. For example, the wearable device (100) can execute a software application (e.g., a software application for a video, a software application for a call, a software application for a game) while providing visual objects for the content (220) and the environment (130) through the display assembly. For example, the wearable device (100) can control at least one first camera (e.g., at least one first camera (309) of FIG. 3) to provide a pass-through function. For example, the wearable device (100) can control at least one second camera (e.g., at least one second camera (310) of FIG. 3) to provide a pass-through function.

[0031] According to one embodiment, the wearable device (100) can provide a visual object for the environment (130) through the display assembly using the pass-through function. For example, a sofa (140) can correspond to a visual object (240). For example, a table (150) can correspond to a visual object (250). For example, an air conditioner (160) can correspond to a visual object (260). For example, a window (170) can correspond to a visual object (270). For example, a refrigerator (180) can correspond to a visual object (280). The wearable device (100) can provide immersiveness to the user (120) by providing at least one visual object (e.g., visual object (240) to visual object (280)) through the display assembly.

[0032] In one embodiment, the power consumed by the wearable device (100) to provide content (220) may be less than the power consumed by the wearable device (100) to provide visual objects (e.g., visual objects (240) to (280)) for the content (220) and the environment (130). For example, the resources required by the wearable device (100) to provide visual objects for the content (220) and the environment (130) may be more than the resources required by the wearable device (100) to provide the content (220). For example, when the user (120) is focused on the content (220), the wearable device (100) may be required to reduce the resources consumed to provide the pass-through function. For example, when a user (120) focuses on content (220), the wearable device (100) may be required to reduce power consumption to provide visual objects corresponding to the environment (130) through the display assembly. For example, the wearable device (100) may perform image processing techniques to provide images of the environment (130) similar to what the user (120) sees through the display assembly (e.g., the display assembly (308) of FIG. 3). For example, the image processing techniques may include resolution enhancement, noise removal, brightness adjustment, and / or color tuning. For example, the image processing techniques may include interpolation of objects to minimize and / or reduce spatial distortion. For example, the image processing techniques may include improving the frame rate of at least one first camera (309). For example, the image processing techniques may include a technique for applying depth values ​​to each of the visual objects included in the images (e.g., visual object (240) to visual object (280)).For example, the wearable device (100) may use at least one processor (e.g., at least one processor (307) of FIG. 3) to perform the image processing techniques. For example, the wearable device (100) may be required to simultaneously execute a plurality of tracking functions (e.g., a head tracking function, a hand tracking function, an eye tracking function, a face tracking function) and a software application for content (220) to provide a pass-through function. For example, the wearable device (100) may use a large amount of resources to provide the pass-through function in which the image processing techniques are performed. For example, the wearable device (100) may use a large amount of power to provide the pass-through function in which the image processing techniques are performed. For example, the wearable device (100) may require a large amount of computation to perform the functions and techniques. For example, a wearable device (100) may be required to reduce resources and power consumed by the wearable device (100).

[0033] For example, the wearable device (100) may change the way it provides the pass-through function when the user (120) focuses on the content (220). For example, the wearable device (100) may provide a screen generated using an image by rendering the image using depth values ​​acquired in relation to an image acquired through at least one first camera (e.g., at least one first camera (309) of FIG. 3) when the user (120) focuses on the environment (130) through the display assembly. For example, the wearable device (100) may provide a screen generated using an image by rendering the image using some of the depth values ​​acquired in relation to the image when the user (120) focuses on the content (220) through the display assembly.

[0034] For example, the wearable device (100) may include hardware components used to perform or execute the above operations. Various hardware components are described and illustrated in more detail with reference to FIG. 3.

[0035] FIG. 3 is a block diagram illustrating an exemplary configuration of an exemplary wearable device according to various embodiments.

[0036] Referring to FIG. 3, the wearable device (100) may include at least one processor (307) (e.g., including a processing circuit), memory (306), a display assembly (308) (e.g., including a processing circuit), at least one first camera (309), and at least one second camera (310).

[0037] At least one processor (307) may include a hardware component for processing data using instructions stored in the memory (306). The hardware component for processing data may include a central processing unit (CPU) (e.g., including processing circuitry). The hardware component for processing data may include a graphic processing unit (GPU) (e.g., including processing circuitry). The hardware component for processing data may include a display processing unit (DPU) (e.g., including processing circuitry). The hardware component for processing data may include a neural processing unit (NPU) (e.g., including processing circuitry).

[0038] According to one embodiment, at least one processor (307) may include one or more cores. For example, at least one processor (307) may have a multi-core processor architecture, such as a dual core, quad core, or hexa core. At least one processor (307) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits, including at least one processor, wherein one or more of the at least one processors may be configured to perform the various functions described herein, individually and / or collectively in a distributed manner. When the terms "processor," "at least one processor," and "one or more processors" as used herein are described as being configured to perform a number of functions, these terms encompass, for example, without limitation, situations where one processor performs some of the recited functions, another processor(s) performs other of the recited functions, and situations where a single processor can perform all of the recited functions. Additionally, the at least one processor may comprise a combination of processors performing various mentioned / disclosed functions, for example, in a distributed manner. At least one processor may execute program instructions to perform or achieve various functions.

[0039] According to one embodiment, the memory (306) may include hardware components for storing data and / or instructions input to and / or output from at least one processor (307). The memory (306) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multimedia card (EMMC).

[0040] According to one embodiment, the display assembly (308) may output visualized information and include a display. For example, the display assembly (308) may output visualized information to a user under the control of at least one processor (307). The display assembly (308) may include hardware components of the wearable device (100) used to display a screen. For example, the display assembly (308) may include light-emitting elements and circuits (e.g., transistors) that control the light-emitting elements to emit light. For example, each of the light-emitting elements may include an organic light emitting diode (OLED) or a micro LED. However, the disclosure is not limited thereto. For example, the display assembly (308) may include a liquid crystal display (LCD).

[0041] According to one embodiment, the display assembly (308) may include a first display positioned in front of the left eye of a user wearing the wearable device (100) and a second display positioned in front of the right eye of the user wearing the wearable device (100). For example, first content provided through a screen displayed through the first display may be (substantially) identical to second content provided through a screen displayed through the second display. Although the first content and the second content are identical, the screen displayed through the second display may have a disparity with respect to the screen displayed through the first display. For example, the disparity may cause the display assembly (308) to provide content (e.g., corresponding to the first content and the second content) in three dimensions.

[0042] As a non-limiting example, the at least one first camera (309) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. For example, the at least one first camera (309) may be described as at least one image sensor. For example, the at least one first camera (309) may be available to acquire an image of an environment surrounding the wearable device (100). For example, the at least one first camera (309) may have a field of view (FOV) corresponding to a user's eye. For example, the at least one first camera (309) may be used to acquire an image of an environment (130) surrounding the wearable device (100). For example, the at least one first camera (309) may be referred to as a Video See-Through (VST) camera.

[0043] As a non-limiting example, the at least one second camera (310) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. For example, the at least one second camera (310) may be described as at least one image sensor. For example, the at least one second camera (310) may be available to acquire an image of an environment surrounding the wearable device (100). For example, the at least one second camera (310) may have a field of view (FOV) corresponding to an FOV of a user's eye. For example, the at least one second camera (310) may be used to acquire an image of an eye of the user (120). For example, the at least one second camera (310) may be arranged with respect to the eye of the user (120).

[0044] As a non-limiting example, at least one processor (307) may identify an event that displays a screen generated using an image acquired through at least one first camera (309). The at least one first camera (309) may be used to acquire an image of the environment (130). The display assembly (308) may include a display area. The at least one processor (307) may display the screen within at least a portion of the display area based on an event that displays the screen within at least a portion of the display area. The at least one processor (307) may display the screen outside at least a portion of the display area based on an event that displays the screen outside at least a portion of the display area. The at least one second camera (310) may be used to acquire an image of an eye of a user (120). The at least one processor (307) may identify the gaze of the user (120) using the image acquired through the at least one second camera (310). At least one processor (307) can display a screen within at least a portion of the display assembly (308) or at least a portion outside the display assembly (308) based on identifying the location of the user's (120) gaze.

[0045] FIG. 4 is a flowchart illustrating an exemplary method for displaying a screen according to various embodiments. This method may be executed by the wearable device (100) illustrated in FIG. 3 or by at least one processor (307) of the wearable device (100).

[0046] Referring to FIG. 4, in operation 410, at least one processor (307) may identify an event that displays a screen generated using an image acquired through at least one first camera (309).

[0047] In operation 420, at least one processor (307) (e.g., a judgment module) may identify a position of a gaze of a user (120). The at least one processor (307) may execute operation 430 under a condition that an event of displaying a screen within at least a portion of a display area where the gaze of the user (120) is located, and may execute operation 440 under a condition that an event of displaying a screen outside at least a portion of the display area where the gaze of the user (120) is located. For example, the display assembly (308) may include a display area. For example, the display area may be described as an area on the display assembly (308) where a screen may be displayed. For example, the user (120) may gaze at at least a portion of the display area. For example, the at least one processor (307) may identify a position of a gaze of the user (120) using an image acquired through at least one second camera (310). For example, at least one processor (307) may identify whether the gaze of the user (120) is located within at least a portion of the display area. For example, at least one processor (307) may determine a region of interest (ROI) based on the location of the gaze of the user (120). For example, at least one processor (307) may determine, based on the gaze of the user (120), that the content related to the gaze of the user (120) is of interest to the user (120). For example, the determination may be performed by at least one processor (307) (e.g., a judgment module).

[0048] In operation 430, at least one processor (307) may display a screen within at least a portion of a display area in which the gaze of the user (120) identified through at least one second camera (310) is located, and may render the image by using depth values ​​acquired in connection with acquiring the image. For example, at least one processor (307) may identify a position of the gaze of the user (120) through at least one second camera (310). For example, at least one processor (307) may identify an event of displaying a screen within at least a portion of the display area. For example, at least one processor (307) may acquire depth values ​​in connection with acquiring an image based on the event. For example, at least one processor (307) may render an image using the acquired depth values. For example, at least one processor (307) may display a screen generated using an image acquired through at least one first camera (309) within at least a portion of the display area, according to rendering the image.

[0049] In operation 440, at least one processor (307) may display the screen outside at least a portion of the display area by rendering an image using some of the depth values ​​based on an event of displaying the screen outside at least a portion of the display area where the gaze identified through at least one second camera (310) is located. For example, at least one processor (307) may identify a position of the gaze of the user (120) through at least one second camera (310). For example, at least one processor (307) may identify an event of displaying the screen outside at least a portion of the display area where the gaze of the user (120) is located. For example, at least one processor (307) may render the image using some of the depth values ​​acquired in connection with acquiring the image. For example, at least one processor (307) may display the screen outside at least a portion of the display area by rendering the image.

[0050] At least one processor (307) may determine at least a portion of the display area where the user's (120) gaze is located as a ROI. For example, the at least one processor (307) may identify at least a portion of the display area where the user's (120) gaze is directed as a ROI based on the user's (120) gaze being located within at least a portion of the display area. For example, the display area may be described as an area on the display assembly (308) where a screen may be displayed. For example, the at least one processor (307) may identify the user's (120) gaze through at least one second camera (310). For example, the at least one processor (307) may perform an eye-tracking function through the at least one second camera (310). For example, the at least one processor (307) may determine that the user (120) is interested in content displayed within at least a portion of the display area based on the user's (120) gaze being directed within at least a portion of the display area. For example, at least one processor (307) may determine that the user (120) is focused on content displayed within at least a portion of the display area based on the user's (120) gaze being directed toward at least a portion of the display area.

[0051] The wearable device (100) may include a sensor (not shown). For example, the sensor may be used to identify the movement of the wearable device (100). For example, the user (120) may move while wearing the wearable device (100). For example, at least one processor (307) may obtain data on the movement of the wearable device (100) through the sensor. For example, at least one processor (307) may determine content of interest to the user (120) using the data on the movement of the wearable device (100). For example, at least one processor (307) may identify the direction in which the wearable device (100) is facing using the data on the movement of the wearable device (100). For example, at least one processor (307) may perform a head-tracking function through the sensor. For example, at least one processor (307) can determine content of interest to the user (120) among the content provided through the display assembly (308) by identifying the direction in which the wearable device (100) is facing. For example, at least one processor (307) can determine content of interest to the user (120) by using an eye tracking function and a head tracking function. For example, at least one processor (307) can determine content of interest to the user (120) among the content provided through the display assembly (308) by using an eye tracking function and a head tracking function.

[0052] According to one embodiment, the wearable device (100) can manage resources by changing the method of rendering images based on an event. For example, the amount of resources consumed may vary depending on the method of rendering images. For example, the resources consumed by the wearable device (100) to provide a pass-through function in a first mode (e.g., the first mode (600) of FIG. 6A) may be greater than the resources consumed by the wearable device (100) to provide a pass-through function in a second mode (e.g., the second mode (605) of FIG. 6A). For example, the first mode (e.g., the first mode (600) of FIG. 6A) may be referred to as a quality mode. For example, the second mode (e.g., the second mode (605) of FIG. 6A) may be referred to as an efficiency mode. For example, the first mode (e.g., the first mode (600) of FIG. 6A) may be described as a mode for operation 430. For example, the second mode (e.g., the second mode (605) of FIG. 6A) may be described as a mode for operation 440. For example, the wearable device (100) may provide a pass-through function in the second mode (e.g., the second mode (605) of FIG. 6A) to provide other functions using the saved resources. For example, the wearable device (100) may provide an eye tracking function or a head tracking function additionally using the saved resources by providing a pass-through function in the second mode (e.g., the second mode (605) of FIG. 6A). For example, the wearable device (100) may provide an pass-through function in the second mode (e.g., the second mode (605) of FIG. 6A) to additionally execute a software application other than a software application for the pass-through function using the saved resources. For example, the wearable device (100) can provide content (220) through the display assembly (308) using the saved resources. For example, the wearable device (100) can maintain pass-through functionality while providing other functions.For example, the wearable device (100) can enhance the immersion of the user (120) by maintaining the pass-through function while providing other functions. For example, the wearable device (100) can increase the efficiency of the system within the wearable device (100) by maintaining the pass-through function while providing other functions.

[0053] At least one processor (307) may control the display (308) to display a screen generated using an image acquired through at least one first camera (309) based on an event, within at least a portion of the display area or outside at least a portion of the display area. For example, the at least one processor (307) may render an image using depth values ​​acquired in connection with acquiring an image through at least one first camera (309). The screen resulting from the image rendering is described and illustrated in more detail with reference to FIGS. 5A and 5B .

[0054] FIGS. 5A and 5B are drawings illustrating examples of rendering images according to various embodiments.

[0055] Referring to FIG. 5A, the screen (510) is provided within at least a portion of the display area where the user's (120) gaze is positioned, and can be described as a screen in which an image is rendered using depth values. For example, displaying the screen in a manner corresponding to the screen (510) can be described as a first mode (e.g., the first mode (600) of FIG. 6A). For example, the first mode (e.g., the first mode (600) of FIG. 6A) can be referred to as a quality mode. For example, at least one processor (307) can display the screen (510) within at least a portion of the display area based on an event that displays the screen (510) within at least a portion of the display area where the user's (120) gaze is positioned. For example, at least one processor (307) can refrain from providing content other than a system user interface (UI) while providing a pass-through function in the first mode. For example, at least one processor (307) may provide a pass-through function and a function related to a system user interface while providing a pass-through function in the first mode. For example, at least one processor (307) may render an image using depth values ​​acquired in relation to an image acquired through at least one first camera (309). For example, at least one processor (307) may render the image by applying depth values ​​for each of a first visual object (e.g., visual object (240)) and a second visual object (e.g., visual object (260)) included in the image to each of the first visual object and the second visual object. For example, at least one processor (307) may acquire depth values ​​for each of visual objects (e.g., visual object (240) to visual object (280)) included in the image while acquiring an image of the environment (130) through at least one first camera (309).For example, the depth values ​​may be used to perform depth reprojection. For example, at least one processor (307) may perform depth reprojection to cause a visual object included in an image to appear as seen by the eyes of the user (120). For example, since the position of at least one first camera (309) of the wearable device (100) is different from the position of the eyes of the user (120), performing depth reprojection may be required to present the visual object to the user (120) through the display assembly (308). The depth reprojection is described and illustrated in more detail with reference to FIG. 5B.

[0056] Referring to FIG. 5B, state (530) can be described as a state in which depth reprojection is performed using depth values ​​on the screen (510). For example, at least one processor (307) can acquire depth values ​​(e.g., depth value (550-1), depth value (550-2), depth value (550-3), depth value (550-4), and / or depth value (550-5)) while acquiring an image through at least one first camera (309). For example, depth value (550-1) can be described as a depth value for a visual object (240). For example, depth value (550-2) can be described as a depth value for a visual object (250). For example, depth value (550-3) can be described as a depth value for a visual object (260). For example, depth value (550-4) may be described as a depth value for visual object (270). For example, depth value (550-5) may be described as a depth value for visual object (280). For example, at least one processor (307) may render an image using depth values. For example, at least one processor (307) may render an image by applying each of the depth values ​​to each of the first visual object and the second visual object. For example, at least one processor (307) may apply depth value (550-1) to visual object (240). For example, at least one processor (307) may apply depth value (550-2) to visual object (250). For example, at least one processor (307) may apply depth value (550-3) to visual object (260). For example, at least one processor (307) can apply a depth value (550-4) to a visual object (270). For example, at least one processor (307) can apply a depth value (550-5) to a visual object (280).For example, at least one processor (307) may render an image using depth values ​​corresponding to each of the visual objects for each of the visual objects, thereby causing the visual objects displayed on the screen (510) to appear as if viewed through the eyes of the user (120).

[0057] Referring back to FIG. 5A, the screen (520) may be described as a screen that is provided outside at least a portion of the display area where the user's (120) gaze is positioned, and in which an image is rendered using some of the depth values. For example, displaying the screen in a manner corresponding to the screen (520) may be described as a second mode (e.g., the second mode (605) of FIG. 6A). For example, the second mode (e.g., the second mode (605) of FIG. 6A) may be referred to as an efficiency mode. For example, at least one processor (307) may display the screen (520) outside at least a portion of the display area based on an event that displays the screen (520) outside at least a portion of the display area where the user's (120) gaze is positioned. For example, the at least a portion of the display area may be described as an area where the user's (120) gaze is positioned. For example, at least one processor (307) may display a high-resolution image on at least a portion of the image while providing a pass-through function in a second mode (e.g., the second mode (605) of FIG. 6A). For example, at least one processor (307) may display a relatively low-resolution image on a different area than at least a portion of the image while providing a pass-through function in a second mode (e.g., the second mode (605) of FIG. 6A). For example, at least one processor (307) may render the image using some of the depth values ​​acquired with respect to an image acquired through at least one first camera (309). For example, at least one processor (307) may render the image by applying a depth value for the first visual object among the depth values ​​for each of a first visual object (e.g., the visual object (240)) and a second visual object (e.g., the visual object (260)) included in the image to each of the first visual object and the second visual object.For example, at least one processor (307) may acquire depth values ​​for each of visual objects (e.g., visual objects (240) to (280)) included in the image while acquiring an image of the environment (130) through at least one first camera (309). For example, at least one processor (307) may perform depth reprojection to cause the visual objects included in the image to appear as if they were seen by the eyes of the user (120). For example, at least one processor (307) may supply power to at least one first camera (309) and at least one second camera (310) while providing a screen (520) through the display assembly (308). For example, at least one processor (307) may provide content (220) through the display assembly (308) while providing a screen (520) through the display assembly (308). For example, at least one processor (307) may be required to control power to perform various functions. For example, at least one processor (307) may be required to manage resources to perform various functions simultaneously. For example, at least one processor (307) may change the manner in which it performs depth reprojection to prevent and / or reduce resource waste while providing a pass-through function. The manner in which it performs depth reprojection is described and exemplified in more detail with reference to FIG. 5B.

[0058] Referring back to FIG. 5B, state (540) may be described as a state in which depth reprojection is performed using some of the depth values ​​on the screen (520). For example, at least one processor (307) may render an image by applying some of the depth values ​​to each of the visual objects included in the image. For example, at least one processor (307) may render an image by applying the depth value (550-1) for the visual object (240) to each of the visual object (240), the visual object (250), the visual object (260), the visual object (270), and the visual object (280). For example, at least one processor (307) may consume power to apply each of the depth values ​​to each of the visual objects. For example, at least one processor (307) may reduce the amount of power consumed to render the image by applying the depth value (550-1) to each of the visual objects. For example, at least one processor (307) may apply a depth value (550-1) to each of the visual objects, thereby reducing the amount of power consumed for rendering the image and freeing up power to provide other functions (e.g., eye tracking, head tracking). For example, if at least one processor (307) performs depth reprojection using some of the depth values, it may be able to provide other functions because the resources consumed for displaying the screen are reduced.

[0059] At least one processor (307) can provide a screen (520) with weak image processing performed by rendering an image using some of the depth values.

[0060] Referring back to FIG. 5A, the screen (520) may be described as a screen displayed by rendering an image using some of the depth values. For example, visual objects within the screen (520) may be described as visual objects that have undergone depth reprojection using some of the depth values. For example, since at least one processor (307) performs depth reprojection using some of the depth values, the visual objects within the screen (520) may appear awkward. For example, the at least one processor (307) may apply a first depth value among a first depth value for a first visual object and a second depth value for a second visual object to the first visual object and the second visual object, so that the visual objects within the screen (520) may not appear as seen by the eyes of the user (120). For example, the visual object (240) within the screen (520) may appear as seen by the eyes of the user (120) by applying a depth value (550-1) to the visual object (240). For example, a visual object (270) within a screen (520) may not appear as seen by the user's (120) eyes by applying a depth value (550-1) to the visual object (240).

[0061] At least one processor (307) may control how depth reprojection is performed to manage resources. For example, at least one processor (307) may control how depth reprojection is performed by adjusting the level of image processing functions or bypassing some of the functional blocks to manage resources. However, the disclosure is not limited thereto. For example, at least one processor (307) may control a denoising function and / or a sharpening function to manage resources. For example, at least one processor (307) may control the FPS (frames per second) and / or the frequency of the sensor to manage resources. For example, at least one processor (307) may control a motion blur removal function to manage resources. For example, at least one processor (307) may control the size of the region for a foveated rendering mode to manage resources.

[0062] According to one embodiment, at least one processor (307) may provide a denoising function using a denoising filter. For example, the denoising function may be described as a function of removing noise from an image. For example, the denoising function may include a blurring function and a filtering function. For example, the blurring function may be described as a function of readjusting a color value of a pixel in an image to a similar color value by comparing the color value of each pixel surrounding the pixel. For example, the filtering function may be described as a function of removing color values ​​that are not necessary for displaying the image by comparing the color value of a pixel in an image to the color values ​​of each pixel surrounding the pixel. For example, at least one processor (307) may perform the denoising function more strongly or to a greater extent in a first mode (e.g., the first mode (600) of FIG. 6A) than in a second mode (e.g., the second mode (605) of FIG. 6A). For example, at least one processor (307) may refrain from or bypass performing the denoising function in the second mode (e.g., the second mode (605) of FIG. 6A). For example, at least one processor (307) may refrain from performing the denoising function in the second mode (e.g., the second mode (605) of FIG. 6A) or may perform it less frequently than in the first mode (e.g., the first mode (600) of FIG. 6A) to manage resources. For example, a denoising filter for performing the denoising function in the first mode (e.g., the first mode (600) of FIG. 6A) may provide a stronger denoising function than another denoising filter for performing the denoising function in the second mode (e.g., the second mode (605) of FIG. 6A).

[0063] According to one embodiment, at least one processor (307) may provide a sharpening function using a sharpening filter. For example, the sharpening function may be described as a function that enhances the sharpness of the screen by emphasizing the boundaries between visual objects included in the image. For example, at least one processor (307) may increase the contrast ratio of the boundaries by performing the sharpening function. For example, at least one processor (307) may cause each visual object to be more easily distinguished by increasing the contrast ratio of the boundaries. For example, at least one processor (307) may maximize and / or increase the difference between the color value of a pixel included in the image and the color values ​​of each of the surrounding pixels by performing the sharpening function. For example, at least one processor (307) may perform the sharpening function more strongly or more frequently in a first mode (e.g., the first mode (600) of FIG. 6A) than in a second mode (e.g., the second mode (605) of FIG. 6A). For example, at least one processor (307) may refrain from performing the sharpening function in the second mode (e.g., the second mode (605) of FIG. 6A) or may perform it less frequently than in the first mode (e.g., the first mode (600) of FIG. 6A) to manage resources. For example, a sharpening filter for performing the sharpening function in the first mode (e.g., the first mode (600) of FIG. 6A) may provide a stronger sharpening function than another sharpening filter for performing the sharpening function in the second mode (e.g., the second mode (605) of FIG. 6A).For example, at least one processor (307) may display the screen (520) at least outside a portion of the display area based on an event that displays the screen outside at least a portion of the display area where the gaze identified through at least one second camera (310) is located, by rendering an image using another denoising filter that performs less denoising than the denoising filter and another sharpening filter that performs less sharpening than the sharpening filter. The at least one processor (307) may perform sharpening and / or denoising. However, the disclosure is not limited thereto. For example, the at least one processor (307) may further perform a function different from sharpening to render the image. For example, the at least one processor (307) may further perform a function different from denoising to render the image.

[0064] At least one processor (307) can control the FPS of a camera (e.g., at least one first camera (309) or at least one second camera (310)) in a first mode (e.g., the first mode (600) of FIG. 6A) to be higher than the FPS of the camera in a second mode (e.g., the second mode (605) of FIG. 6A). For example, the at least one processor (307) can control the FPS of the camera to manage resources because the higher the FPS of the camera, the more power it consumes. The at least one processor (307) can control the frequency of a sensor (not shown) in the first mode (e.g., the first mode (600) of FIG. 6A) to be higher than the frequency of the sensor in a second mode (e.g., the second mode (605) of FIG. 6A). For example, the at least one processor (307) can control the frequency of the sensor to be higher than the frequency of the sensor in a second mode (e.g., the second mode (605) of FIG. 6A). For example, at least one processor (307) can obtain a more accurate depth value (550) as the frequency of the sensor increases. For example, at least one processor (307) can obtain a depth value by using the ToF (Time of Flight) technique through the sensor. For example, the ToF technique may include the dToF (direct Time of Flight) technique and the iToF (indirect Time of Flight) technique. For example, the ToF technique can be described as a technique that identifies the difference in wavelength and phase when the reflected light reaches the sensor after the emitted light reaches the object. The dToF technique can be described as a technique that calculates the distance between the object and the sensor by using the speed of light. The iToF technique can be described as a technique that calculates the distance between the object and the sensor by analyzing the phase difference of the light.

[0065] At least one processor (307) may perform motion blur removal less or less strongly in the second mode (e.g., the second mode (605) of FIG. 6A) than in the first mode (e.g., the first mode (600) of FIG. 6A). For example, motion blur may be described as an afterimage of a visual object contained within a video frame appearing on the display. For example, the at least one processor (307) may insert a black screen between video frames to remove motion blur. For example, the at least one processor (307) may perform motion blur removal less strongly in the second mode (e.g., the second mode (605) of FIG. 6A) than in the first mode (e.g., the first mode (600) of FIG. 6A) to manage resources.

[0066] According to one embodiment, at least one processor (307) may perform image processing or change a mode based on an area ratio of the display assembly (308). For example, at least one processor (307) may change a mode from a second mode (e.g., the second mode (605) of FIG. 6A) to a first mode (e.g., the first mode (600) of FIG. 6A) based on a determination that a ratio of a screen displayed through the display assembly (308) and indicating a pass-through function exceeds a threshold ratio. For example, at least one processor (307) may change a mode from a second mode (e.g., the second mode (605) of FIG. 6A) to a first mode (e.g., the first mode (600) of FIG. 6A) based on a determination that a ratio of a screen indicating a virtual object is less than a threshold ratio. For example, at least one processor (307) can identify a ratio of an area on which an image representing a pass-through and another image representing a virtual object are displayed on the display assembly (308). For example, at least one processor (307) can identify a type of screen displayed through the display assembly (308). For example, at least one processor (307) can identify whether a screen displayed through the display assembly (308) is a screen representing a pass-through function. For example, at least one processor (307) can identify whether a screen displayed through the display assembly (308) is a screen representing a virtual reality (VR) environment. For example, at least one processor (307) can identify a ratio of a display area of ​​the display assembly (308) of a screen representing a pass-through function displayed through the display assembly (308). For example, at least one processor (307) can identify a ratio of a display area of ​​the display assembly (308) of a screen representing a VR environment displayed through the display assembly (308).For example, at least one processor (307) may perform denoising and / or sharpening on the screen based on a determination that a percentage of the display area of ​​the display assembly (308) of the screen representing the pass-through function exceeds a threshold percentage (e.g., 40%).

[0067] At least one processor (307) may take different amounts of time to render an image depending on how the screen is presented. For example, the time taken to render an image in a first mode (e.g., the first mode (600) of FIG. 6A) may be longer than the time taken to render an image in a second mode (e.g., the second mode (605) of FIG. 6A). For example, since at least one processor (307) performs at least one of a denoising function, a sharpening function, and a motion blur removal function more strongly or more frequently than each of the functions performed in the second mode (e.g., the second mode (605) of FIG. 6a) to render an image in the first mode (e.g., the first mode (600) of FIG. 6a), the time required to render an image in the first mode (e.g., the first mode (600) of FIG. 6a) may be longer than the time required to render an image in the second mode (e.g., the second mode (605) of FIG. 6a).

[0068] According to one embodiment, at least one processor (307) can change the mode for rendering an image based on a user input. For example, the user input may include receiving through an input device. For example, the user input may include detecting a gaze of the user (120). For example, the user input may include detecting a gaze of the user (120) toward an executable object (not shown) for a change in mode. For example, the wearable device (100) may include an input device (not shown). For example, the wearable device (100) may receive a user input through the input device. For example, the wearable device (100) may change the mode for rendering an image from a first mode (e.g., the first mode (600) of FIG. 6A) to a second mode (e.g., the second mode (605) of FIG. 6A) based on a user input received through the input device. For example, the wearable device (100) may change the mode for rendering an image from a second mode (e.g., the second mode (605) of FIG. 6A) to a first mode (e.g., the first mode (600) of FIG. 6A) based on a user input received through the input device. However, the disclosure is not limited thereto. For example, the wearable device (100) may change the mode for rendering an image based on detecting a gaze of a user (120) toward an executable object (not shown) displayed through the display assembly (308). For example, the wearable device (100) may render an image by using depth values ​​acquired in connection with acquiring an image before receiving a user input, thereby providing a screen (510) through the display assembly (308).For example, the wearable device (100) may provide a screen (520) through the display assembly (308) by rendering an image using some of the acquired depth values ​​in connection with acquiring an image based on receiving a user input.

[0069] At least one processor (307) may provide a pass-through function in a first mode (e.g., the first mode (600) of FIG. 6A) or a second mode (e.g., the second mode (605) of FIG. 6A) based on the state of the user (120). The first mode and the second mode are described and illustrated in more detail with reference to FIG. 6A.

[0070] FIG. 6A is a diagram illustrating an example explaining the characteristics of each mode according to various embodiments.

[0071] Referring to FIG. 6A, at least one processor (307) may provide a pass-through function to the user (120) that operates in a first mode (600) or a second mode (605). For example, at least one processor (307) may identify whether the user (120) is focused on content (220). For example, at least one processor (307) may execute the pass-through function in the second mode (605) based on identifying that the user (120) is focused on content (220). For example, at least one processor (307) may provide the pass-through function in the first mode (600) based on identifying that the user (120) is not focused on content (220). For example, the first mode (600) and the second mode (605) may operate according to priorities for image processing techniques.

[0072] The first mode (600) may be referred to as a quality mode. For example, at least one processor (307) may operate with high priority for peripheral comfort, full-view comfort, distortion and artifact reduction, and image quality in the first mode (600). For example, at least one processor (307) may operate with low priority for performance in the second mode (605). For example, the peripheral comfort and full-view comfort may be referred to as reprojection for depth values ​​as described in FIGS. 5A and 5B . For example, the peripheral comfort may be described in a manner in which depth value correction is applied to the periphery of the screen displayed through the display assembly (308). For example, the periphery may include a different area from the area in which the content (220) is displayed. For example, the full-view comfort may be described in a manner in which depth value correction is applied to the entire screen displayed through the display assembly (308). The distortion and artifact reduction function may include a function for reducing spatial distortion. For example, the distortion and artifact reduction function may be performed by performing interpolation on images acquired through at least one first camera (309). The image quality may include descriptions of denoising, sharpening, and / or motion blur as described with reference to FIG. 5A. For example, the performance may include performance for all of the functions provided by the wearable device (100). For example, the performance may include performance for a function different from the pass-through function. For example, performance for pass-through in the first mode (600) may be better than performance for pass-through in the second mode (605). For example, performance for playback of content (220) in the first mode (600) may be worse than performance for playback of content (220) in the second mode (605).

[0073] The second mode (605) may be referred to as an efficiency mode. For example, at least one processor (307) may operate with high priority for peripheral comfort, distortion and artifact reduction, and performance in the second mode (605). For example, at least one processor (307) may operate with low priority for full-view comfort and image quality in the second mode (605).

[0074] Mode 1 (Quality Mode) Mode 2 (Efficiency Mode) VST Camera 90 fps 90 fpsi ToF speed and dToF speed Less than 2-5 fps Less than 2 fps VST Resolution Phobias: Less than 1500x1500 Peripherals: 1500x1500 Phobias are much smaller than 1500x1500. VST resolution is less than 1500x1500 P2P and M2PP 2P is less than 20-25 ms M2P is less than 20 ms P2P is less than 20-25 ms M2P is less than 20 ms IQ (image quality) GPU Budget Less than 5 ms Less than 2 ms

[0075] Referring to Table 1, the performance of the first mode (600) and the second mode (605) is described. For example, P2P (point-to-point) can be described as a direct data transfer method between two points. For example, M2P (memory-to-processor) can be described as a method of transferring data from memory to a processor. For example, the first mode (600) can operate at a higher resolution than the second mode (605). At least one processor (307) can provide a pass-through function in the first mode (600) or the second mode (605). For example, at least one processor (307) can switch the mode in which the pass-through function is provided from the first mode (600) to the second mode (605) based on a user input. For example, at least one processor (307) can switch the mode in which the pass-through function is provided from the second mode (605) to the first mode (600) based on the state of the user (120).

[0076] First stateActionLater stateWearable device is offTurn on wearable deviceHome screen in 1st mode or 2nd modeHome screen in 2nd modeGive a double tap to wearable device1st mode1st modeGive a double tap to wearable deviceReturn to previous state1st modeRecenterReturn to home screen in 2nd modeOpen home screenReturn to home screen in 2nd modeHome screen in 2nd modeGo out of guardian area1st modeOut of guardian area1st modeEnter guardian areaReturn to previous stateHome screen in VRGive a swipe to wearable deviceHome screen in 2nd modeHome screen in 2nd modeGive input to passthrough input deviceHome screen in VRHome screen in 2nd modeGive a swipe to wearable deviceHome screen in VR1st modeGive a swipe to wearable deviceUnresponsiveLaunch a software application on VRGive a swipe to wearable deviceUnresponsiveLaunch a software application on VRGive a double tap to wearable device1st modeLaunch a software application on VRGo out of guardian area1st mode

[0077] Referring to Table 2, it is described that the first mode (600) is switched to the second mode (605) or the second mode (605) is switched to the first mode (600) based on an action of the user (120). For example, the initial state of Table 2 may be described as a state before an action occurs. For example, the later state of Table 2 may be described as a state after an action occurs. For example, the action may include an input or action provided by the user (120) while in the initial state. For example, the wearable device (100) may be switched from the initial state to the later state based on identifying the action. At least one processor (307) may provide a screen, through the display assembly (308), in the first mode or the second mode (605) based on identifying an event. For example, at least one processor (307) may provide a screen in a first mode by approaching the boundary of a recommended area (e.g., recommended area (610)) in which activity of the wearable device (100) is recommended. For example, the recommended area is described and illustrated in more detail with reference to FIG. 6B.

[0078] FIG. 6b is a diagram illustrating examples of events related to recommended areas according to various embodiments.

[0079] Referring to FIG. 6B, at least one processor (307) can identify whether the electronic device (100) is within a recommended area (610). For example, the at least one processor (307) can identify the recommended area (610) using an image acquired through at least one first camera (309). For example, the at least one processor (307) can identify whether the wearable device (100) is within the recommended area (610) using the image. For example, the recommended area (610) can be described as an area where the movement of the user (120) is recommended for the safety of the user (120). For example, the recommended area (610) can be referred to as a guardian area. For example, the recommended area (610) can be set in the shape of a circle centered on the user (120). However, the disclosure is not limited thereto. For example, the recommended area (610) may be set in various shapes (e.g., polygons) considering the environment (130) including the user (120). For example, at least one processor (307) may provide a screen, through a display assembly (308) including a display area, in a first mode (600) or a second mode (605) based on an event. For example, the event may include the wearable device (100) approaching a boundary of a recommended area (610) recommended for the movement of the user (120). For example, when the user (120) approaches the boundary of the recommended area (610), at least one processor (307) may be required to provide a pass-through function to prevent and / or reduce an accident of the user (120). For example, at least one processor (307) may measure or determine the distance between the wearable device (100) and the boundary of the recommended area (610) using an image acquired through at least one first camera (309).For example, at least one processor (307) may determine that the user (120) is interested in the environment (130) by identifying that the user (120) is at the boundary of the recommended area (610). For example, while identifying that the user (120) is at the boundary of the recommended area (610), the at least one processor (307) may provide a pass-through function to the first mode (600) among the first mode (600) and the second mode (605). For example, the at least one processor (307) may display the screen in the first mode (600) based on a determination that the distance between the boundary of the recommended area (610) and the wearable device (100) is less than a threshold distance. For example, at least one processor (307) may display a screen (510) within at least a portion of the display area by rendering an image using depth values ​​acquired in connection with acquiring an image based on an event in which the distance between the user (120) and the boundary of the recommended area (610) is less than a threshold distance. For example, at least one processor (307) may display a screen within at least a portion of the display area in which the gaze of the user (120) identified through at least one second camera (310) is located, and may display a screen (510) within at least a portion of the display area based on an event in which the distance between the user (120) of the wearable device (100) and the boundary of the recommended area (610) recommended for the movement of the user (120) is less than a threshold distance. For example, at least one processor (307) may identify whether the user (120) is within the recommended area (610). For example, at least one processor (307) may provide a pass-through function in the first mode (600) or the second mode (605) based on identifying whether the user (120) is within a recommended area (610).For example, at least one processor (307) may use images acquired through at least one first camera (309) to identify whether the user (120) is within the recommended area (610). For example, at least one processor (307) may provide a pass-through function in the first mode (600) based on a determination that the user (120) is outside the recommended area (610). For example, at least one processor (307) may provide a pass-through function in the second mode (605) based on a determination that the user (120) is within the recommended area (610).

[0080] For example, at least one processor (307) may perform alpha blending based on determining that the wearable device (100) is positioned at the boundary of the recommendation area (610). For example, the alpha blending may be described as a technique for blending images by controlling the transparency of different images displayed through the display assembly (308). For example, at least one processor (307) may perform alpha blending on an image representing a pass-through and another image representing a virtual screen based on determining that the wearable device (100) is positioned at the boundary of the recommendation area (610). For example, performing alpha blending on an image representing a pass-through and another image representing a virtual screen may consume a lot of power. For example, at least one processor (307) may lower the resolution of the image representing a pass-through to reduce the power consumed for performing alpha blending. For example, at least one processor (307) may disable image processing for an image representing passthrough or reduce the intensity of image processing to reduce power consumption for performing alpha blending. For example, the image processing may include at least one of a denoising function, a sharpening function, and a motion blur removal function. For example, at least one processor (307) may change the mode of the wearable device (100) from a second mode (605) to a first mode (600) based on a determination that the wearable device (100) is located at the boundary of a recommended area (610).

[0081] For example, at least one processor (307) can provide content (220) through the display assembly (308). For example, at least one processor (307) can identify the location of the user's (120) gaze through at least one second camera (310). For example, at least one processor (307) can detect the user's (120) gaze toward the content (220) through at least one second camera (310). For example, at least one processor (307) can display the screen in the second mode (605) based on the user's (120) gaze being located outside at least a portion of the display area where the user's (120) gaze is located for a preset period of time while the content (220) is provided outside the at least portion of the display area where the user's (120) gaze is located. The detection of the gaze for the content is described and exemplified in more detail with reference to FIG. 7.

[0082] FIG. 7 is a diagram illustrating examples of events related to content according to various embodiments.

[0083] Referring to FIG. 7, a state (710) can be described as a state in which visual objects corresponding to content (220) and environment (130) are displayed through a display assembly (308). At least one processor (307) can display a screen (510) within at least a portion of the display area based on an event of displaying a screen within at least a portion of the display area while the content (220) is displayed within at least a portion of the display area where a gaze identified through at least one second camera (310) is located, by rendering an image using depth values ​​(e.g., depth value (550-1) to depth value (550-5)). At least one processor (307) may display the screen (520) outside at least a portion of the display area by rendering an image using a portion (e.g., depth value (550-1)) of depth values ​​(e.g., depth value (550-1) to depth value (550-5)) based on an event that displays the screen outside at least a portion of the display area while the content (220) is displayed outside at least a portion of the display area where the gaze identified through at least one second camera (310) is located. For example, the at least one processor (307) may identify the position of the gaze of the user (120) using an image acquired through the at least one second camera (310). For example, the at least one processor (307) may detect the gaze of the user (120) through the at least one second camera (310). For example, at least one processor (307) can identify at least a portion of a display area included in the display assembly (308) through at least one second camera (310) where the user's (120's) gaze is located. For example, at least a portion of the display area can be described as an area where the user's (120's) gaze is located. For example, the user (120) can gaze at the content (220) or another screen while the content (220) is provided through the display assembly (308).For example, at least one processor (307) may determine that the user (120) is interested in the content (220) based on receiving user input for playing the content (220). For example, at least one processor (307) may provide a pass-through function to the second mode (605) based on a determination that the user (120) is interested in the content (220). For example, at least one processor (307) may render an image using some of the depth values ​​based on a determination that the user (120) is interested in the content (220), thereby displaying the screen (520) at least a portion outside the display area.

[0084] However, the disclosure is not limited thereto. For example, at least one processor (307) may detect a gaze directed toward an area other than the area in which the content (220) is played while the content (220) is provided through the display assembly (308). For example, at least one processor (307) may detect a gaze of the user (120) toward one of the visual object (240), the visual object (250), the visual object (260), the visual object (270), and the visual object (280) while the content (220) is provided through the display assembly (308). For example, at least one processor (307) may determine that the user (120) is interested in the environment (130) based on detecting the gaze directed toward the other area. For example, at least one processor (307) may provide a pass-through function to the first mode (600) based on a determination that the user (120) is interested in the environment. For example, at least one processor (307) may render an image using depth values ​​to display the screen (510) within at least a portion of the display area.

[0085] At least one processor (307) may provide a foveated rendering mode while providing a pass-through function. For example, at least one processor (307) may control the area of ​​the foveated rendering mode differently depending on the first mode (600) or the second mode (605). The foveated rendering mode is described and exemplified in more detail with reference to FIG. 8.

[0086] FIG. 8 is a diagram illustrating an example of controlling the size of an area related to a foveated rendering mode according to various embodiments.

[0087] Referring to FIG. 8, state (810) can be described as a state in which a high-resolution area of ​​the foveated rendering mode changes depending on the first mode (600) or the second mode (605). For example, the foveated rendering mode can be described as a mode that provides screens with different resolutions by distinguishing the display area of ​​the display assembly (308). For example, at least one processor (307) can determine the central portion of the display area as a high-resolution area and determine the peripheral portion of the display area as a low-resolution area while the foveated rendering mode is executed. For example, at least one processor (307) can display an image processed with a high resolution in the high-resolution area of ​​the display area. For example, at least one processor (307) can display an image processed with a low resolution in the low-resolution area of ​​the display area. For example, at least one processor (307) can reduce power consumed to provide an image by distinguishing between the high-resolution area and the low-resolution area. For example, at least one processor (307) can manage resources by distinguishing between high-resolution areas and low-resolution areas.

[0088] At least one processor (307) may determine the sizes of the high-resolution area and the low-resolution area of ​​the foveated rendering mode differently depending on the first mode (600) or the second mode (605). For example, the at least one processor (307) may display the screen (510) within at least a portion of the display area by rendering the image using depth values ​​acquired in connection with acquiring the image based on an event of displaying the screen within at least a portion of the display area where the gaze of the user (120) identified through at least one second camera (310) is located. For example, the first mode (600) may be described as rendering the image using depth values ​​acquired in connection with acquiring the image. For example, the at least one processor (307) may display the screen (520) outside at least a portion of the display area by rendering the image using some of the depth values ​​based on an event of displaying the screen outside at least a portion of the display area where the gaze of the user (120) identified through at least one second camera (310) is located. For example, the second mode (605) may be described as rendering an image using a portion of the acquired depth values ​​(e.g., depth value (550-1)) in relation to acquiring an image. For example, the power consumed by the wearable device (100) while applying the first mode (600) may be more than the power consumed by the wearable device (100) while applying the second mode (605). For example, since the first mode (600) is intended to provide a high quality image or screen to the user (120), at least one processor (307) may provide a large size of a high resolution area while the foveated rendering mode is provided in the first mode (600).

[0089] For example, at least one processor (307) may determine the region (820) as a high-resolution region of the foveated rendering mode while the second mode (605) is applied. For example, based on the region (820) being determined as a high-resolution region, at least one processor (307) may provide a high-quality image or screen in the region (820) and provide a low-quality image or screen in a region other than the region (e.g., region (830) to region (840)).

[0090] For example, at least one processor (307) may determine the region (830) as a high-resolution region of the foveated rendering mode while the first mode (600) is applied. For example, based on the region (830) being determined as a high-resolution region, the at least one processor (307) may provide a high-quality image or screen in the region (830) and provide a low-quality image or screen in a region other than the region (e.g., region (840)). For example, in order to provide a high-quality image or screen to the user (120) in the first mode (600), the size of the high-resolution region in the first mode (600) may be larger than the size of the high-resolution region in the second mode (605). For example, the high-resolution region of the foveated rendering mode in the first mode (600) may be region (830). For example, the high-resolution region of the foveated rendering mode in the second mode (605) may be region (820).

[0091] FIG. 9 is a block diagram illustrating an exemplary electronic device within a network environment according to various embodiments.

[0092] Referring to FIG. 9, in a network environment (900), an electronic device (901) may communicate with an electronic device (902) via a first network (998) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (904) or a server (908) via a second network (999) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (901) may communicate with the electronic device (904) via the server (908). According to one embodiment, the electronic device (901) may include a processor (920), a memory (930), an input module (950), an audio output module (955), a display module (960), an audio module (970), a sensor module (976), an interface (977), a connection terminal (978), a haptic module (979), a camera module (980), a power management module (988), a battery (989), a communication module (990), a subscriber identification module (996), or an antenna module (997). In various embodiments, the electronic device (901) may omit at least one of these components (e.g., the connection terminal (978)), or may have one or more other components added. In various embodiments, some of these components (e.g., the sensor module (976), the camera module (980), or the antenna module (997)) may be integrated into one component (e.g., the display module (960)).

[0093] The processor (920) may include various processing circuits and / or multiple processors. For example, the term "processor," as used herein, including in the claims, may include various processing circuits, including at least one processor, wherein one or more of the at least one processor may be configured to perform various functions described herein, individually and / or collectively in a distributed manner. When the terms "processor," "at least one processor," and "one or more processors," as used herein, are described as being configured to perform a number of functions, these terms encompass, for example, without limitation, a situation where one processor performs some of the recited functions, another processor(s) performs other of the recited functions, and a situation where a single processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform various recited / disclosed functions, for example, in a distributed manner. At least one processor may execute program instructions to perform or accomplish various functions. The processor (920) may, for example, execute software (e.g., a program (940)) to control at least one other component (e.g., a hardware or software component) of the electronic device (901) connected to the processor (920) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (920) may store commands or data received from other components (e.g., a sensor module (976) or a communication module (990)) in a volatile memory (932), process the commands or data stored in the volatile memory (932), and store result data in a non-volatile memory (934).According to one embodiment, the processor (920) may include a main processor (921) (e.g., a central processing unit or an application processor) or an auxiliary processor (923) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (921). For example, when the electronic device (901) includes the main processor (921) and the auxiliary processor (923), the auxiliary processor (923) may be configured to use less power than the main processor (921) or to be specialized for a given function. The auxiliary processor (923) may be implemented separately from the main processor (921) or as a part thereof.

[0094] The auxiliary processor (923) may control at least a portion of functions or states associated with at least one component (e.g., a display module (960), a sensor module (976), or a communication module (990)) of the electronic device (901), for example, on behalf of the main processor (921) while the main processor (921) is in an inactive (e.g., sleep) state, or together with the main processor (921) while the main processor (921) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (923) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (980) or a communication module (990)). In one embodiment, the auxiliary processor (923) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (901) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (908)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0095] The memory (930) can store various data used by at least one component (e.g., the processor (920) or the sensor module (976)) of the electronic device (901). The data can include, for example, software (e.g., the program (940)) and input data or output data for commands related thereto. The memory (930) can include a volatile memory (932) or a non-volatile memory (934).

[0096] The program (940) may be stored as software in the memory (930) and may include, for example, an operating system (942), middleware (944), or an application (946).

[0097] The input module (950) can receive commands or data to be used in a component of the electronic device (901) (e.g., a processor (920)) from an external source (e.g., a user) of the electronic device (901). The input module (950) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0098] The audio output module (955) can output audio signals to the outside of the electronic device (901). The audio output module (955) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0099] The display module (960) can visually provide information to an external party (e.g., a user) of the electronic device (901). The display module (960) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (960) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0100] The audio module (970) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (970) can acquire sound through the input module (950), output sound through the sound output module (955), or an external electronic device (e.g., electronic device (902)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (901).

[0101] The sensor module (976) can detect the operating status (e.g., power or temperature) of the electronic device (901) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (976) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0102] The interface (977) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (901) with an external electronic device (e.g., the electronic device (902)). In one embodiment, the interface (977) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0103] The connection terminal (978) may include a connector through which the electronic device (901) may be physically connected to an external electronic device (e.g., the electronic device (902)). In one embodiment, the connection terminal (978) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0104] The haptic module (979) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (979) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0105] The camera module (980) can capture still images and videos. According to one embodiment, the camera module (980) may include one or more lenses, image sensors, image signal processors, or flashes.

[0106] The power management module (988) can manage the power supplied to the electronic device (901). According to one embodiment, the power management module (988) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0107] A battery (989) may power at least one component of the electronic device (901). In one embodiment, the battery (989) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0108] The communication module (990) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (901) and an external electronic device (e.g., electronic device (902), electronic device (904), or server (908)), and the performance of communication through the established communication channel. The communication module (990) may operate independently from the processor (920) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (990) may include a wireless communication module (992) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (994) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (904) via a first network (998) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (999) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (992) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (996) to identify or authenticate the electronic device (901) within a communication network such as the first network (998) or the second network (999).

[0109] The wireless communication module (992) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (992) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (992) can support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (992) can support various requirements specified in the electronic device (901), an external electronic device (e.g., the electronic device (904)), or a network system (e.g., the second network (999)). According to one embodiment, the wireless communication module (992) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0110] The antenna module (997) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (997) may include an antenna including a radiator including a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (997) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (998) or the second network (999), may be selected from the plurality of antennas by, for example, the communication module (990). A signal or power may be transmitted or received between the communication module (990) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (997).

[0111] According to various embodiments, the antenna module (997) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0112] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0113] According to one embodiment, commands or data may be transmitted or received between the electronic device (901) and an external electronic device (904) via a server (908) connected to a second network (999). Each of the external electronic devices (902 or 904) may be the same or a different type of device as the electronic device (901). According to one embodiment, all or part of the operations executed in the electronic device (901) may be executed in one or more of the external electronic devices (902, 904, or 908). For example, when the electronic device (901) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (901) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (901). The electronic device (901) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (901) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In an embodiment, the external electronic device (904) may include an Internet of Things (IoT) device. The server (908) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (904) or the server (908) may be included in the second network (999).The electronic device (901) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0114] FIG. 10A is a perspective view illustrating an exemplary wearable device according to various embodiments.

[0115] FIG. 10b is a perspective view illustrating an example of one or more hardware elements arranged within a wearable device according to various embodiments.

[0116] According to one embodiment, a wearable device (100) (e.g., an electronic device (901)) may have a form of glasses that can be worn on a body part (e.g., a head) of a user. The electronic device (901) of FIGS. 10A and 10B may be an example of the wearable device (100) of FIG. 1. The wearable device (100) may include a head-mounted display (HMD). For example, the housing of the wearable device (100) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (100) may include one or more straps that can be twined around the user's head, and / or one or more temples that can be detachably attached to the ears of the head.

[0117] Referring to FIG. 10A, according to one embodiment, a wearable device (100) may include at least one display (1050) and a frame (1000) supporting at least one display (1050).

[0118] According to one embodiment, the wearable device (100) can be worn on a part of a user's body. The wearable device (100) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (100). For example, the wearable device (100) can display a virtual reality image provided from at least one optical device (1082, 1084) of FIG. 10b on at least one display (1050) in response to a user's designated gesture acquired through the motion recognition cameras (1060-2, 1060-3) of FIG. 10b.

[0119] According to one embodiment, at least one display (1050) may provide visual information to a user. For example, at least one display (1050) may include a transparent or translucent lens. At least one display (1050) may include a first display (1050-1) and / or a second display (1050-2) spaced apart from the first display (1050-1). For example, the first display (1050-1) and the second display (1050-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0120] Referring to FIG. 10B, at least one display (1050) can provide a user with visual information transmitted from external light and other visual information distinct from the visual information through a lens included in the at least one display (1050). The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (1050) can include a first surface (1031) and a second surface (1032) opposite the first surface (1031). A display area can be formed on the second surface (1032) of the at least one display (1050). When a user wears the wearable device (100), external light can be transmitted to the user by being incident on the first surface (1031) and transmitted through the second surface (1032). As another example, at least one display (1050) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (1082, 1084) on a real screen transmitted through external light, in a display area formed on the second surface (1032).

[0121] In one embodiment, at least one display (1050) may include at least one waveguide (1033, 1034) that diffracts light emitted from at least one optical device (1082, 1084) and transmits the diffracted light to a user. The at least one waveguide (1033, 1034) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (1033, 1034). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (1033, 1034) may be propagated to the other end of the at least one waveguide (1033, 1034) by the nano-pattern. At least one waveguide (1033, 1034) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1033, 1034) may be arranged within the wearable device (100) to guide a screen displayed by at least one display (1050) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (1033, 1034).

[0122] The wearable device (100) can analyze an object included in a real image collected through a shooting camera (1060-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1050). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (100) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (100) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (100) can view an image displayed on at least one display (1050).

[0123] According to one embodiment, the frame (1000) may be formed as a physical structure that allows the wearable device (100) to be worn on the user's body. According to one embodiment, the frame (1000) may be configured so that, when the user wears the wearable device (100), the first display (1050-1) and the second display (1050-2) can be positioned corresponding to the user's left and right eyes. The frame (1000) may support at least one display (1050). For example, the frame (1000) may support the first display (1050-1) and the second display (1050-2) to be positioned corresponding to the user's left and right eyes.

[0124] Referring to FIG. 10A, the frame (1000) may include a region (1020) that at least partially contacts a portion of the user's body when the user wears the wearable device (100). For example, the region (1020) of the frame (1000) that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (100) touches. According to one embodiment, the frame (1000) may include a nose pad (1010) that contacts a portion of the user's body. When the wearable device (100) is worn by the user, the nose pad (1010) may contact a portion of the user's nose. The frame (1000) may include a first temple (1004) and a second temple (1005) that contact another part of the user's body that is distinct from the part of the user's body.

[0125] For example, the frame (1000) may include a first rim (1001) that surrounds at least a portion of the first display (1050-1), a second rim (1002) that surrounds at least a portion of the second display (1050-2), a bridge (1003) that is disposed between the first rim (1001) and the second rim (1002), a first pad (1011) that is disposed along a portion of the edge of the first rim (1001) from one end of the bridge (1003), a second pad (1012) that is disposed along a portion of the edge of the second rim (1002) from the other end of the bridge (1003), a first temple (1004) that extends from the first rim (1001) and is fixed to a portion of the ear of the wearer, and a second temple (1005) that extends from the second rim (1002) and is fixed to a portion of the ear opposite the ear. There are. The first pad (1011) and the second pad (1012) can be in contact with a part of the user's nose, and the first temple (1004) and the second temple (1005) can be in contact with a part of the user's face and a part of the user's ear. The temples (1004, 1005) can be rotatably connected to the rim through the hinge units (1006, 1007) of FIG. 10B. The first temple (1004) can be rotatably connected to the first rim (1001) through the first hinge unit (1006) disposed between the first rim (1001) and the first temple (1004). The second temple (1005) may be rotatably connected to the second rim (1002) via a second hinge unit (1007) disposed between the second rim (1002) and the second temple (1005). In one embodiment, the wearable device (100) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (1000) to identify an external object (e.g., a user's fingertip) touching the frame (1000) and / or a gesture performed by the external object.

[0126] According to one embodiment, the wearable device (100) may include hardwares (e.g., hardwares described above based on the block diagram of FIG. 4) that perform various functions. For example, the hardwares may include a battery module (1070), an antenna module (1075), at least one optical device (1082, 1084), speakers (e.g., speakers 1055-1, 1055-2), a microphone (e.g., microphones 1065-1, 1065-2, 1065-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (1090) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (1000).

[0127] According to one embodiment, microphones (e.g., microphones 1065-1, 1065-2, 1065-3) of the wearable device (100) may be disposed on at least a portion of the frame (1000) to acquire sound signals. A first microphone (1065-1) disposed on the bridge (1003), a second microphone (1065-2) disposed on the second rim (1002), and a third microphone (1065-3) disposed on the first rim (1001) are illustrated in FIG. 10B , but the number and arrangement of the microphones (1065) are not limited to the embodiment of FIG. 10B . When the number of microphones (1065) included in the wearable device (100) is two or more, the wearable device (100) can identify the direction of a sound signal by using a plurality of microphones placed on different parts of the frame (1000).

[0128] According to one embodiment, at least one optical device (1082, 1084) may project a virtual object onto at least one display (1050) to provide various image information to a user. For example, at least one optical device (1082, 1084) may be a projector. At least one optical device (1082, 1084) may be disposed adjacent to at least one display (1050) or may be included within at least one display (1050) as a part of at least one display (1050). According to one embodiment, the wearable device (100) may include a first optical device (1082) corresponding to a first display (1050-1) and a second optical device (1084) corresponding to a second display (1050-2). For example, at least one optical device (1082, 1084) may include a first optical device (1082) disposed at an edge of a first display (1050-1) and a second optical device (1084) disposed at an edge of a second display (1050-2). The first optical device (1082) may transmit light to a first waveguide (1033) disposed on the first display (1050-1), and the second optical device (1084) may transmit light to a second waveguide (1034) disposed on the second display (1050-2).

[0129] In one embodiment, the camera (1060) may include a recording camera (1060-4), an eye tracking camera (ET CAM) (1060-1), and / or a motion recognition camera (1060-2, 1060-3). The recording camera (1060-4), the eye tracking camera (1060-1), and the motion recognition cameras (1060-2, 1060-3) may be positioned at different locations on the frame (1000) and may perform different functions. The eye tracking camera (1060-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (100). For example, the wearable device (100) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (1060-1). The wearable device (100) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (1060-1). The wearable device (100) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (100) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (1060-1). The wearable device (100) can render an image (or screen) displayed on at least one display (1050) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other.The wearable device (100) can obtain an image having a visual quality of a first region and a visual quality of a second region that match the user's gaze using foveated rendering. For example, if the wearable device (100) supports an iris recognition function, user authentication can be performed based on iris information obtained using a gaze tracking camera (1060-1). Although an example in which the gaze tracking camera (1060-1) is positioned toward the user's right eye is illustrated in FIG. 10B, the disclosure is not limited thereto, and the gaze tracking camera (1060-1) can be positioned solely toward the user's left eye, or toward both eyes.

[0130] In one embodiment, the capturing camera (1060-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (1060-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (1060-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1050). The at least one display (1050) can display a single image in which information about an actual image or background including an image of the specific object acquired using the capturing camera (1060-4) and a virtual image provided through at least one optical device (1082, 1084) are superimposed. The wearable device (100) can compensate for depth information (e.g., the distance between the wearable device (100) and an external object acquired through a depth sensor) using an image acquired through the capturing camera (1060-4). The wearable device (100) can perform object recognition using an image acquired using the capturing camera (1060-4). The wearable device (100) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capturing camera (1060-4). The wearable device (100) can perform a pass-through function to display an image acquired through the capturing camera (1060-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (1050) while displaying the screen. In one embodiment, the shooting camera (1060-4) may be positioned on a bridge (1003) positioned between the first rim (1001) and the second rim (1002).

[0131] The gaze tracking camera (1060-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (100) and matching the user's gaze with visual information provided to at least one display (1050). For example, when the wearable device (100) looks straight ahead, the wearable device (100) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1050). The gaze tracking camera (1060-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1060-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1060-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1060-1) may be positioned within the first rim (1001) and / or the second rim (1002) to face the direction in which the user wearing the wearable device (100) is positioned.

[0132] The gesture recognition camera (1060-2, 1060-3) can provide a specific event on a screen provided on at least one display (1050) by recognizing the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the user's body. The gesture recognition camera (1060-2, 1060-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1050). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (1060-2, 1060-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (1060-2, 1060-3). In one embodiment, the motion recognition cameras (1060-2, 1060-3) may be positioned on the first rim (1001) and / or the second rim (1002).

[0133] The camera (1060) included in the wearable device (100) is not limited to the above-described gaze tracking camera (1060-1) and motion recognition cameras (1060-2, 1060-3). For example, the wearable device (100) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (100) can identify an external object based on a sensor for identifying the distance between the wearable device (100) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1060) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (100) may include a camera (1060) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (100).

[0134] Although not shown, in one embodiment, the wearable device (100) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being captured using the camera (1060). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame (1000) and the hinge units (1006, 1007).

[0135] According to one embodiment, the battery module (1070) may supply power to electronic components of the wearable device (100). In one embodiment, the battery module (1070) may be disposed within the first temple (1004) and / or the second temple (1005). For example, the battery module (1070) may be a plurality of battery modules (1070). The plurality of battery modules (1070) may be disposed within each of the first temple (1004) and the second temple (1005). In one embodiment, the battery module (1070) may be disposed at an end of the first temple (1004) and / or the second temple (1005).

[0136] The antenna module (1075) includes at least one antenna and can transmit signals or power to the outside of the wearable device (100), or receive signals or power from the outside. In one embodiment, the antenna module (1075) can be positioned within the first temple (1004) and / or the second temple (1005). For example, the antenna module (1075) can be positioned close to one surface of the first temple (1004) and / or the second temple (1005).

[0137] The speaker (1055) can output an audio signal to the outside of the wearable device (100). The audio output module may be referred to as a speaker. In one embodiment, the speaker (1055) may be positioned within the first temple (1004) and / or the second temple (1005) so as to be positioned adjacent to the ear of a user wearing the wearable device (100). For example, the speaker (1055) may include a second speaker (1055-2) positioned within the first temple (1004) and thus adjacent to the user's left ear, and a first speaker (1055-1) positioned within the second temple (1005) and thus adjacent to the user's right ear.

[0138] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (100) to the user. For example, when the wearable device (100) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (1001) and / or the second rim (1002).

[0139] Referring to FIG. 10B, according to one embodiment, a wearable device (100) may include a printed circuit board (PCB) (1090). The PCB (1090) may be included in at least one of the first temple (1004) or the second temple (1005). The PCB (1090) may include an interposer positioned between at least two sub-PCBs. One or more hardwares included in the wearable device (100) (e.g., hardwares illustrated by different blocks in FIG. 4) may be positioned on the PCB (1090). The wearable device (100) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0140] According to one embodiment, a wearable device (100) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (100) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (100). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (100) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (100) based on the IMU.

[0141] FIGS. 11A and 11B are perspective views illustrating examples of the appearance of a wearable device according to various embodiments.

[0142] The electronic device (901) of FIGS. 11A and 11B may be an example of the wearable device (100) of FIG. 1. According to one embodiment, an example of the appearance of a first side (1110) of a housing of the wearable device (100) is illustrated in FIG. 11A, and an example of the appearance of a second side (1120) opposite to the first side (1110) may be illustrated in FIG. 11B.

[0143] Referring to FIG. 11A, a first surface (1110) of a wearable device (100) according to one embodiment may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (100) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1004) and / or the second temple (1005) of FIGS. 10A and 10B). A first display (1050-1) for outputting an image to a left eye among the user's two eyes, and a second display (1050-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (1110). The wearable device (100) may further include a rubber or silicone packing formed on the first surface (1110) to prevent and / or reduce interference by light (e.g., ambient light) different from the light emitted from the first display (1050-1) and the second display (1050-2).

[0144] According to one embodiment, the wearable device (100) may include cameras (1060-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (1050-1) and the second display (1050-2). The cameras (1060-1) may be referred to as the gaze tracking camera (1060-1) of FIG. 10B. According to one embodiment, the wearable device (100) may include cameras (1060-5, 1060-6) for photographing and / or recognizing the face of the user. The cameras (1060-5, 1060-6) may be referred to as FT cameras. The wearable device (100) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (1060-5, 1060-6). For example, the wearable device (100) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar expressing a human face) using information obtained by cameras (1060-5, 1060-6) (e.g., FT cameras) and representing a facial expression of a user wearing the wearable device (100).

[0145] Referring to FIG. 11b, a camera (e.g., cameras (1060-7, 1060-8, 1060-9, 1060-10, 1060-11, 1060-12)) and / or a sensor (e.g., a depth sensor (1130)) for obtaining information related to the external environment of the wearable device (100) may be disposed on a second surface (1120) opposite to the first surface (1110) of FIG. 11a. For example, the cameras (1060-7, 1060-8, 1060-9, 1060-10) may be disposed on the second surface (1120) for recognizing external objects. Cameras (1060-7, 1060-8, 1060-9, 1060-10) may be referenced to the motion recognition cameras (1060-2, 1060-3) of FIG. 10b.

[0146] For example, using cameras (1060-11, 1060-12), the wearable device (100) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1060-11) can be placed on the second face (1120) of the wearable device (100) to obtain an image to be displayed through the second display (1050-2) corresponding to the right eye among the two eyes. The camera (1060-12) can be placed on the second face (1120) of the wearable device (100) to obtain an image to be displayed through the first display (1050-1) corresponding to the left eye among the two eyes. The cameras (1060-11, 1060-12) can be referred to as the shooting camera (1060-4) of FIG. 10B.

[0147] According to one embodiment, the wearable device (100) may include a depth sensor (1130) disposed on the second face (1120) to identify a distance between the wearable device (100) and an external object. Using the depth sensor (1130), the wearable device (100) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (100). Although not illustrated, a microphone may be disposed on the second face (1120) of the wearable device (100) to obtain a sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0148] Hereinafter, with reference to FIG. 12, the hardware or software configuration of the wearable device (100) is described.

[0149] FIG. 12 is a block diagram illustrating an exemplary configuration of a wearable device according to various embodiments.

[0150] The wearable device (100) of FIG. 12 may be an example of the wearable device (100) of FIG. 1 and the electronic device (901) of FIGS. 10A to 11B.

[0151] Referring to FIG. 12, a wearable device (100) according to one embodiment may include a processor (1210) (e.g., including a processing circuit), a memory (1215), a display (1050) (e.g., the first display (1050-1) and / or the second display (1050-2) of FIGS. 10A, 10B, 11A, and 11B), and / or a sensor (1220). The processor (1210), the memory (1215), the display (1050), and / or the sensor (1220) may be electrically and / or operatively connected to each other by electronic components, such as a communication bus (1202). In the present disclosure, the operative connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (100) is not limited to those illustrated in FIG. 12. For example, the wearable device (100) may include only some of the electronic components illustrated in FIG. 12.

[0152] According to one embodiment, a processor (1210) of a wearable device (100) may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (100) may include one or more processors. The processor (1210) may have a multi-core processor structure such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (1210) may include a structure (e.g., a big-little structure) based on a plurality of core circuits that are distinguished by power consumption, clock, and / or calculation amount per unit time. In one embodiment comprising a processor (1210) having a multi-core processor architecture, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (1210).

[0153] According to one embodiment, the memory (1215) of the wearable device (100) may include electronic components for storing data and / or instructions input to and / or output from the processor (1210). The memory (1215) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC). In one embodiment, memory (1215) may be referred to as storage.

[0154] In one embodiment, a display (1050) of a wearable device (100) can output visualized information to a user of the wearable device (100). The display (1050), which is arranged in front of the eyes of a user wearing the wearable device (100), can be arranged on at least a portion of a housing of the wearable device (100) (e.g., the first display (1050-1) and / or the second display (1050-2) of FIGS. 10A, 10B, 11A, and 11B). For example, the display (1050) can be controlled by a processor (1210) including circuits such as a CPU, a GPU (graphics processing unit), and / or a DPU (display processing unit), to output visualized information to the user. The display (1050) can include a flexible display, a FPD (flat panel display), and / or electronic paper. The display (1050) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). The disclosure is not limited thereto, and for example, if the wearable device (100) includes a lens for transmitting external light (or ambient light), the display (1050) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (1050) may be referred to as a display panel and / or a display module. The pixels included in the display (1050) may be arranged to face either of the user's eyes when the wearable device (100) is worn by the user.For example, the display (1050) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0155] In one embodiment, the sensor (1220) of the wearable device (100) may generate electrical information that may be processed by the processor (1210) and / or the memory (1215) from non-electronic information related to the wearable device (100). For example, the sensor (1220) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (100). In addition to the GPS method, the sensor (1220) may generate information indicating the geographic location of the wearable device (100) based on a global navigation satellite system (GNSS) such as, for example, Galileo or Beidou (compass). The above information may be stored in memory (1215), processed by processor (1210), and / or transmitted to another electronic device distinct from the wearable device (100) via communication circuitry.

[0156] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (1210) of the wearable device (100) may be stored in the memory (1215) of the wearable device (100). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (100) and / or the processor (1210) may perform at least one of the operations of FIG. 4 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, a program, and / or a software application is executed. Hereinafter, the fact that a software application is installed in a wearable device (100) may mean that one or more instructions provided in the form of a software application (or package) are stored in a memory (1215), and that the one or more applications are stored in a format executable by the processor (1210) (e.g., a file having an extension specified by the operating system of the wearable device (100)). For example, an application may include a program and / or a library related to a service provided to a user.

[0157] Referring to FIG. 12, programs installed in the wearable device (100) may be included in any one of different layers, including an application layer (1240), a framework layer (1250), and / or a hardware abstraction layer (HAL) (1280), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (1050), and / or the sensor (1220)) of the wearable device (100) may be included in the hardware abstraction layer (1280). The framework layer (1250) may be referred to as an XR framework layer from the perspective of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 12 may be logically (or for convenience of explanation) separated, and may not mean that the address space of the memory (1215) is separated by the layers.

[0158] For example, within the framework layer (1250), programs designed to target at least one of the hardware abstraction layer (1280) and / or the application layer (1240) (e.g., a position tracker (1271), a space recognizer (1272), a gesture tracker (1273), an eye-gaze tracker (1274), and / or a face tracker (1275)) may be included. The programs included in the framework layer (1250) may provide an application programming interface (API) that is executable (or callable) based on other programs.

[0159] For example, a program designed to target users of a wearable device (100) may be included within the application layer (1240). As an example of programs included in the application layer (1240), an extended reality (XR) system user interface (UI) (1241) and / or an XR application (1242) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (1240) may call an API to cause execution of functions supported by programs included in the framework layer (1250).

[0160] For example, the wearable device (100) may display one or more visual objects on the display (1050) for performing interaction with the user based on the execution of the XR system UI (1241). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (100) may provide the user with functions available within a virtual space based on the execution of the XR system UI (1241).

[0161] Referring to FIG. 12, a lightweight renderer (1243) and / or an XR plug-in (1244) are illustrated to be included within the XR system UI (1241), but are not limited thereto. For example, based on the XR system UI (1241), the processor (1210) may execute a lightweight renderer (1243) and / or an XR plug-in (1244) within the framework layer (1250).

[0162] For example, the wearable device (100) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (1243). The lightweight renderer (1243) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (1243) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (100) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (1244). The XR plugin (1244) can be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0163] For example, the wearable device (100) may display a screen representing at least a portion of a virtual space on the display (1050) based on the execution of the XR application (1242). The XR plug-in (1244-1) included in the XR application (1242) may include instructions that support functions similar to those of the XR plug-in (1244) of the XR system UI (1241). Descriptions of the XR plug-in (1244-1) that overlap with those of the XR plug-in (1244) may be omitted. The wearable device (100) may cause the execution of the virtual space manager (1251) based on the execution of the XR application (1242).

[0164] For example, the wearable device (100) may display an image on the display (1050) in a virtual space based on the execution of the application (1245). The application (1245) may be configured to output image information for displaying a two-dimensional image. The wearable device (100) may cause the execution of the virtual space manager (1251) based on the execution of the application (1245). The wearable device (100) may generate dual image information to display the two-dimensional image in a three-dimensional virtual space based on the execution of the application (1245). Here, the dual image information may include first image information for the left eye and second image information for the right eye in consideration of binocular disparity. In order to display the two-dimensional image in the three-dimensional virtual space, the wearable device (100) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0165] According to one embodiment, the wearable device (100) may provide a virtual space service based on the execution of the virtual space manager (1251). For example, the virtual space manager (1251) may include a platform for supporting the virtual space service. The wearable device (100) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (1230) based on the execution of the virtual space manager (1251), and may display at least a portion of the virtual space on the display (1050). The virtual space manager (1251) may be referred to as a composition presentation manager (CPM).

[0166] For example, the virtual space manager (1251) may include a runtime service (1252). As an example, the runtime service (1252) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (100) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (1252). As an example, the wearable device (100) may perform rendering for a virtual space service for the user based on the execution of the runtime service (1252). For example, a function related to a virtual space, executable by the application layer (1240), may be supported based on the execution of the runtime service (1252).

[0167] For example, the virtual space manager (1251) may include a pass-through manager (1253). Based on the execution of the pass-through manager (1253), the wearable device (100) may display an image and / or video representing an actual space acquired through an external camera on at least a portion of the screen while displaying a screen representing a virtual space on the display (1050).

[0168] For example, the virtual space manager (1251) may include an input manager (1254). The wearable device (100) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (1270) based on the execution of the input manager (1254). The wearable device (100) may use the acquired data to identify user input related to the wearable device (100). The user input may be related to a motion (e.g., a hand gesture), gaze, and / or speech of the user identified by a sensor (1220) (e.g., an image sensor (1230) such as an external camera). The user input may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0169] For example, the perception abstract layer (1260) can be used for data exchange between the virtual space manager (1251) and the perception service layer (1270). From the perspective of being used for data exchange between the virtual space manager (1251) and the perception service layer (1270), the perception abstract layer (1260) can be referred to as an interface. For example, the perception abstract layer (1260) can be referenced as OpenPX. The perception abstract layer (1260) can be used for a perception client and a perception service.

[0170] According to one embodiment, the recognition service layer (1270) may include one or more programs for processing data acquired from the sensor (1220). The one or more programs may include at least one of a position tracker (1271), a space recognizer (1272), a gesture tracker (1273), and / or an eye tracker (1274). The type and / or number of the one or more programs included in the recognition service layer (1270) are not limited to those illustrated in FIG. 12.

[0171] For example, the wearable device (100) can identify the pose of the wearable device (100) using the sensor (1230) based on the execution of the position tracker (1271). The wearable device (100) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (100) using data acquired using an external camera (e.g., an image sensor (1221)) and / or an IMU (e.g., a motion sensor (1222) including a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (1271). The position tracker (1271) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0172] For example, the wearable device (100) may obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (100) (or the user of the wearable device (100)) based on the execution of the space recognizer (1272). The wearable device (100) may reproduce the surrounding environment of the wearable device (100) in three dimensions using data obtained using an external camera (e.g., an image sensor (1221)) based on the execution of the space recognizer (1272). The wearable device (100) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (100) reproduced in three dimensions based on the execution of the space recognizer (1272). The space recognizer (1272) may be referred to as a scene understanding (SU) module (or a scene recognition program).

[0173] For example, the wearable device (100) may identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (100) based on the execution of the gesture tracker (1273). As an example, the wearable device (100) may identify a pose and / or gesture of a hand of a user using data acquired from an external camera (e.g., an image sensor (1221)) based on the execution of the gesture tracker (1273). As an example, the wearable device (100) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using an external camera based on the execution of the gesture tracker (1273). The gesture tracker (1273) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0174] For example, the wearable device (100) may identify (or track) eye movements of a user of the wearable device (100) based on the execution of the gaze tracker (1274). As an example, the wearable device (100) may identify eye movements of the user using data acquired from a gaze tracking camera (e.g., an image sensor (1221)) based on the execution of the gaze tracker (1274). The gaze tracker (1274) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0175] For example, the recognition service layer (1270) of the wearable device (100) may further include a face tracker (1275) for tracking the user's face. For example, the wearable device (100) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (1275). The wearable device (100) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (1275). As an example, the wearable device (100) may identify the movement of the user's face and / or the user's expression based on data (e.g., images and / or videos) acquired using a camera (1225) (e.g., a camera facing at least a portion of the user's face) based on the execution of the face tracker (1275).

[0176] Referring to FIG. 12, the renderer (1290) may include instructions for rendering images in a three-dimensional virtual space. The processor (1210) executing the renderer (1290) may obtain at least one image to be at least partially displayed in the display area of ​​the display (1050) in a software application. For example, the processor (1210) executing the renderer (1290) may determine the location of the area where an application (e.g., XR application (1242), application (1245)) is to be rendered. The processor (1210) executing the renderer (1290) may generate an image of the application to be displayed on the display (1050). The renderer (1290) may synthesize images to generate a composite image to be displayed on the display (1050).

[0177] For example, the processor (1210) executing the renderer (1290) can divide the display area of ​​the display (1050) into a foveated portion (or may be referred to as the foveated area) and a peripheral portion (or may be referred to as the residual area) using the gaze position calculated using the position tracker (1271) and / or the gaze tracker (1274). For example, the processor (1210) detecting the coordinate values ​​of the gaze position can determine the portion of the display area including the coordinate values ​​as the foveated area. The DPU executing the renderer (1290) can obtain at least one image corresponding to each of the foveated area and the residual area, and having a size smaller than the size of the entire display area of ​​the display (1050) or a resolution smaller than the resolution of the display area.

[0178] The processor (1210) executing the renderer (1290) may obtain or generate a composite image to be displayed on the display (1050) by synthesizing an image corresponding to the foveated area and an image corresponding to the surrounding area. For example, the processor (1210) may perform upscaling to enlarge the image corresponding to the surrounding area to the size of the entire display area of ​​the display (1050). On the enlarged image, the processor (1210) may combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (1050). Along the boundary line of the image corresponding to the foveated area, the processor (1210) may apply a visual effect, such as blur, to blend the enlarged image and the image corresponding to the foveated area.

[0179] FIG. 13 is a block diagram illustrating an exemplary configuration of an electronic device for displaying an image in a virtual space according to various embodiments.

[0180] In Fig. 13, an example is described in which multiple programs / instructions are executed to display an image in a virtual space. The multiple programs / instructions may be executed entirely on a single processor (e.g., an AP) or by multiple processors (e.g., an AP, a GPU (graphics processing unit), or an NPU (neural processing unit)). Being able to be executed by multiple processors means that some programs / instructions may be executed by a first processor, and other programs / instructions may be executed by a second processor different from the first processor.

[0181] Referring to FIG. 13, the electronic device (901) may execute a virtual space manager (1350) (e.g., the virtual space manager (1251) of FIG. 12, CPM) to render an image in a virtual space. For the virtual space manager (1350), at least some of the descriptions of the virtual space manager (1251) of FIG. 12 may be referenced. The virtual space manager (1350) may include a platform for supporting a virtual space service. The virtual space manager (1350) may include a runtime service (1351) (e.g., OpenXR Runtime), a panel renderer (1352) (e.g., 2D Panel Render), and an XR compositor (1353). The electronic device (901) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (1351). For the runtime service (1351), at least some of the descriptions of the runtime service (1252) of FIG. 12 may be referred to. The electronic device (901) may display at least one image (video) on a panel (e.g., a 2D panel) to implement a virtual space through the display based on the execution of the panel rendering (1352). For example, the electronic device (901) may display a rendering image corresponding to RGB information (1366) for the panel from the spatialization manager (1340) described below through the display (e.g., the display (1050)). The electronic device (901) may synthesize an image of an actual area captured by a camera in the virtual space (hereinafter, a pass-through image) with a virtual area image based on the execution of the XR compositor (1353). For example, the electronic device (901) can generate a composite image by merging the pass-through image and the virtual area image based on the execution of the XR synthesis unit (1353).The electronic device (901) may transmit the generated composite image to a display buffer so that the composite image is displayed. The electronic device (901) may identify a virtual space through a virtual space manager (1350) and display at least a portion of the virtual space on the display (1050). The virtual space manager (1350) may be referred to as a CPM. The electronic device (901) may execute the virtual space manager (1350) to render an image corresponding to at least a portion of the virtual space.

[0182] According to one embodiment, the electronic device (901) may execute a spatialization manager (1340). The spatialization manager (1340) may perform processes for displaying an image in a three-dimensional virtual space. The electronic device (901) may perform preprocessing based on the execution of the spatialization manager (1340) so that the image can be rendered in a three-dimensional virtual space through the virtual space manager (1350). For example, the electronic device (901) may perform at least some of the functions of the renderer (1290) of FIG. 12 based on the execution of the spatialization manager (1340). The electronic device (901) may process image information provided by an application (e.g., an XR application (1310), an application (1320) that provides a general 2D screen other than XR, and an application that provides a system UI (1330)) based on the execution of the spatialization manager (1340). A spatialization manager (1340) (e.g., Space Flinger) may include a system scene manager (1341) (e.g., System scene), an input manager (1342) (e.g., Input Routing), and a lightweight rendering engine (1343) (e.g., Impress Engine). The system scene manager (1341) may be executed to display a system UI (1330). System UI-related information (1364) may be transmitted to the system scene manager (1341) from a program (e.g., API) that provides the system UI (1330). The system UI-related information (1364) may be obtained through a spatializer API and / or a same-process private API. The spatialization manager (1340) may determine the layout (e.g., location, display order) of the screen of the system UI (1330) in a three-dimensional space through pre-allocated resources.The system screen manager (1341) may transmit image information (1367) for rendering the screen of the system UI (1330) to the virtual space manager (1350) according to the layout. The input manager (1342) may be configured to process user input (e.g., user input on a system screen or an app screen). The impression engine (1343) may be a renderer for image generation (e.g., a lightweight renderer (1243)). For example, the impression engine (1343) may be used to display the system UI (1330). According to one embodiment, the spatialization manager (1340) may include a lightweight rendering engine (1343) for rendering the system UI. According to one embodiment, when the lightweight rendering engine (1343) does not have sufficient resources to render an avatar used in the HMD, at least one external rendering engine may be used. At this time, to resolve compatibility issues with external rendering (e.g., 3rd party engines), an external rendering engine support module may be added within the spatialization manager (1340).

[0183] According to one embodiment, the electronic device can execute an application. For example, in response to the execution of an XR application (1310) (e.g., an XR application (1242), a 3D game, an XR map, or other immersive application), the electronic device can execute a virtual space manager (1350). The electronic device (901) can provide dual image information (1361) provided from the XR application (1310) to the virtual space manager (1350). In order to display an image in a three-dimensional space, the dual image information (1361) can include two pieces of image information that take binocular parallax into account. For example, the dual image information (1361) can include first image information for the user's left eye and second image information for the user's right eye for rendering in a three-dimensional virtual space. Hereinafter, in the present disclosure, the term dual image information is used to refer to image information for displaying images for both eyes in a three-dimensional space. In addition to the dual image information, the above dual image information may also include binocular image information, dual image information, dual image data, dual images, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 10D-3D conversion data, dimensional conversion image data, binocular parallax image data, and / or equivalent technical terms. The electronic device (901) can generate a composite image by merging image layers through the virtual space manager (1350). The electronic device (901) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (1050) of the electronic device (901).

[0184] According to one embodiment, the electronic device can execute at least one application among an XR application (1310) and other applications (1320) (e.g., a first application (1320-1), a second application (1320-2), ..., an Nth application (1320-N)). According to one embodiment, the application (1320) can be configured to output image information for displaying a two-dimensional image. In other words, the application (1320) can provide a two-dimensional image. For example, the application (1320) can be a video application, a schedule application, or an application (1320) can be an Internet browser application. If it is assumed that in response to the execution of the application (1320), image information (1362) provided from the application (1320) is provided to the virtual space manager (1350). Since the image information (1362) only has x-coordinates and y-coordinates within a two-dimensional plane, it may be difficult to consider the chronological relationship (i.e., the distance from the user) between other applications centered on the user. Even when displaying an application (1320) that provides a general 2D screen, the electronic device (901) may execute the spatialization manager (1340) to provide dual image information to the virtual space manager (1350). For example, based on the execution of the spatialization manager (1340), the electronic device (901) may receive application-related information (1363) from the first application (1320-1). For example, the application-related information (1363) may include image information representing a two-dimensional image of the first application (1320-1) (e.g., information including RGB for each pixel) and / or content information in the first application (1320-1) (e.g., characteristics of content executed in the first application, type of content). Application related information (1363) can be obtained through the spatializer API.Based on the execution of the spatialization manager (1340), the electronic device (901) can identify information about the location of the area to be rendered by the first application (1320-1) and the size of the area to be rendered (hereinafter, location information). Based on the execution of the spatialization manager (1340), the electronic device (901) can generate dual image information (1365, e.g., RGBx2) that takes into account the user's binocular disparity through the image information and the location information. Based on the execution of the spatialization manager (1340), the electronic device (901) can provide the dual image information (1365) to the virtual space manager (1350). By converting a simple two-dimensional image into the dual image information (1365), a problem that occurs when the image information (1362) is directly transmitted to the virtual space manager (1350) can be resolved. Additionally, since at least some of the functions for displaying images in a virtual space are performed by the spatialization manager (1340) instead of the virtual space manager (1350), the burden on the virtual space manager (1350) can be reduced.

[0185] The wearable device as described above may include a memory (e.g., memory (306)) that stores instructions. The wearable device may include at least one first camera (e.g., at least one first camera (309)). The wearable device may include at least one second camera (e.g., at least one second camera (310)). The wearable device may include a display assembly (e.g., display assembly (308)) that includes at least one display including a display area. The wearable device may include at least one processor (e.g., at least one processor (307)). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify an event for displaying a screen generated using an image acquired through the at least one first camera. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen within at least a portion of the display area, based on the event for displaying the screen within at least a portion of the display area where the gaze of a user (e.g., user (120)) identified through the at least one second camera is located, by using depth values ​​(e.g., depth value (550-1) to depth value (550-5)) acquired in connection with acquiring the image.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen outside the at least a portion of the display area by rendering the image using a portion of the depth values ​​based on the event to display the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera is located.

[0186] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen within the at least a portion of the display area by rendering the image by applying a first depth value (e.g., depth value (550-1)) for a first visual object (e.g., visual object (240)) included in the image to the first visual object and applying a second depth value (e.g., depth value (550-2)) for a second visual object (e.g., visual object (250)) included in the image to the second visual object, based on the event for displaying the screen within the at least a portion of the display area where the gaze of the user identified through the at least one second camera is located. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen outside the at least a portion of the display area by rendering the image by applying the first depth value of the first visual object and the second depth value of the second visual object to the first visual object and the second visual object based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera is located.

[0187] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen within the at least a portion of the display area by further using a denoising filter configured to remove noise from the screen and a sharpening filter configured to enhance sharpness of the screen, based on the event for displaying the screen within the at least a portion of the display area where the gaze of the user identified through the at least one second camera is located. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen outside the at least a portion of the display area by further rendering the image using another denoising filter configured to perform less denoising than the denoising filter and another sharpening filter configured to perform less sharpening than the sharpening filter, based on the event to display the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera is located.

[0188] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen within at least a portion of the display area by rendering the image using the depth values ​​acquired in connection with acquiring the image based on the event that a distance between the user of the wearable device and a boundary of a recommended area (e.g., recommended area (610)) recommended for movement of the user is less than a threshold distance.

[0189] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the screen outside the at least part of the display area by rendering the image using the portion of the depth values ​​based on the event to display the screen outside the at least part of the display area while the content (e.g., content (220)) is displayed outside the at least part of the display area where the gaze identified through the at least one second camera is located.

[0190] In one embodiment, the wearable device may further include a sensor configured to detect movement of the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain movement data of the wearable device through the sensor. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify the event for displaying the screen generated using the movement data and the image obtained through the at least one first camera.

[0191] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to reduce a frames per second (FPS) of the at least one first camera to reduce power consumed by the at least one first camera based on the event to display the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera is located.

[0192] A method performed by a wearable device (e.g., wearable device (100)) having at least one first camera (e.g., at least one first camera (309)), at least one second camera (e.g., at least one second camera (310)), and a display assembly (e.g., display assembly (308)) including at least one display having a display area, as described above, may include an operation of identifying an event for displaying a screen generated using an image acquired through the at least one first camera. The method may include an operation of displaying the screen within the at least a portion of the display area by rendering the image using depth values ​​acquired in connection with acquiring the image (e.g., depth values ​​(550-1) to (550-5)), based on the event for displaying the screen within at least a portion of the display area where a gaze of a user (e.g., user (120)) identified through the at least one second camera is located. The method may include an action of displaying the screen outside the at least part of the display area by rendering the image using a portion of the depth values ​​based on the event for displaying the screen outside the at least part of the display area where the gaze identified through the at least one second camera is located.

[0193] According to one embodiment, the method may include an operation of displaying the screen within at least a portion of the display area by rendering the image by applying a first depth value (e.g., depth value (550-1)) for a first visual object (e.g., visual object (240)) included in the image to the first visual object and applying a second depth value (550-2) for a second visual object (e.g., visual object (250)) included in the image to the second visual object, based on the event for displaying the screen within at least a portion of the display area where the gaze of the user identified through the at least one second camera is located. The method may include an action of displaying the screen outside the at least a portion of the display area by rendering the image by applying the first depth value among the first depth value for the first visual object and the second depth value for the second visual object to the first visual object and the second visual object based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera is located.

[0194] In one embodiment, the method may include displaying the screen within the at least part of the display area, based on the event for displaying the screen within the at least part of the display area where the gaze of the user identified through the at least one second camera is located, by further using a denoising filter configured to remove noise from the screen and a sharpening filter configured to improve sharpness of the screen, thereby rendering the image. The method may include displaying the screen outside the at least part of the display area, based on the event for displaying the screen outside the at least part of the display area where the gaze of the user identified through the at least one second camera is located, by further using another denoising filter configured to perform less denoising than the denoising filter and another sharpening filter configured to perform less sharpening than the sharpening filter, thereby rendering the image.

[0195] According to one embodiment, the method may include an action of displaying the screen within at least a portion of the display area by rendering the image using the depth values ​​acquired in connection with acquiring the image based on the event that a distance between the user of the wearable device and a boundary of a recommended area (e.g., recommended area (610)) recommended for movement of the user is less than a threshold distance.

[0196] In one embodiment, the method may include displaying the screen outside the at least part of the display area by rendering the image using the portion of the depth values ​​based on the event for displaying the screen outside the at least part of the display area while the content (e.g., content (220)) is displayed outside the at least part of the display area where the gaze identified through the at least one second camera is located.

[0197] According to one embodiment, the wearable device may further include a sensor configured to detect movement of the wearable device. The method may include an operation of acquiring movement data of the wearable device through the sensor. The method may include an operation of identifying the event for displaying the screen generated using the movement data and the image acquired through the at least one first camera.

[0198] According to one embodiment, the method may include an operation of reducing the frames per second (FPS) of the at least one first camera to reduce power consumed by the at least one first camera based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera is located.

[0199] In a computer-readable storage medium having one or more programs stored thereon, as described above, the one or more programs may include instructions that, when executed by a wearable device (e.g., a wearable device (100)) having at least one first camera (e.g., at least one first camera (309)), at least one second camera (e.g., at least one second camera (310)), and a display assembly (e.g., a display assembly (308)) including at least one display including a display area, cause the wearable device to identify an event for displaying a screen generated using an image acquired through the at least one first camera. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the screen within at least a portion of the display area in which the gaze of a user (e.g., user (120)) identified through the at least one second camera is located, by rendering the image using depth values ​​(e.g., depth value (550-1) to depth value (550-5)) acquired in connection with acquiring the image based on the event for displaying the screen within at least a portion of the display area in which the gaze of the user (e.g., user (120)) identified through the at least one second camera is located. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the screen outside the at least portion of the display area in which the gaze of the user (e.g., user (120)) identified through the at least one second camera is located, by rendering the image using a portion of the depth values.

[0200] According to one embodiment, the one or more programs may include instructions that cause the wearable device to display the screen within at least a portion of the display area by rendering the image by applying a first depth value (e.g., depth value (550-1)) for a first visual object (e.g., visual object (240)) included in the image to the first visual object and applying a second depth value (e.g., depth value (550-2)) for a second visual object (e.g., visual object (250)) included in the image to the second visual object, based on the event for displaying the screen within at least a portion of the display area where the gaze of the user identified through the at least one second camera is located, when executed by the wearable device. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the screen outside the at least a portion of the display area by rendering the image by applying the first depth value of the first visual object and the second depth value of the second visual object to the first visual object and the second visual object based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera is located.

[0201] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the screen within the at least a portion of the display area based on the event for displaying the screen within the at least a portion of the display area where the gaze of the user identified through the at least one second camera is located, by further using a denoising filter configured to remove noise from the screen and a sharpening filter configured to enhance sharpness of the screen, thereby rendering the image. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the screen outside the at least a portion of the display area by further rendering the image using another denoising filter configured to perform less denoising than the denoising filter and another sharpening filter configured to perform less sharpening than the sharpening filter, based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera is located.

[0202] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the screen within at least a portion of the display area by rendering the image using the depth values ​​acquired in connection with acquiring the image based on the event that a distance between the user of the wearable device and a boundary of a recommended area (e.g., recommended area (610)) recommended for movement of the user is less than a threshold distance.

[0203] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the screen outside the at least part of the display area by rendering the image using the portion of the depth values ​​based on the event to display the screen outside the at least part of the display area while the content (e.g., content (220)) is displayed outside the at least part of the display area where the gaze identified through the at least one second camera is located.

[0204] In one embodiment, the wearable device may further include a sensor configured to detect movement of the wearable device. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to obtain movement data of the wearable device through the sensor. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify the event for displaying the screen generated using the movement data and the image obtained through the at least one first camera.

[0205] In one embodiment, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to reduce a frames per second (FPS) of the at least one first camera to reduce power consumed by the at least one first camera based on the event to display the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera is located.

[0206] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in various embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0207] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0208] The method according to the exemplary embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0209] Although the various embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0210] Therefore, other implementations and various embodiments are included within the scope of the present disclosure, including the appended claims and their equivalents. It will also be appreciated that any one of the embodiments described herein may be utilized with any other embodiment described herein. In one embodiment, the method according to the various embodiments disclosed herein may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0211] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (100), A memory that stores instructions and includes one or more storage media; At least one first camera (309); At least one second camera (310); a display assembly (308) comprising at least one display including a display area; and At least one processor comprising processing circuitry, The above instructions, when individually or collectively executed by the at least one processor, Identify an event for displaying a screen generated using an image acquired through at least one first camera (309), Based on the event for displaying the screen within at least a portion of the display area where the gaze of the user (120) identified through the at least one second camera (310) is located, the image is rendered using depth values ​​acquired in connection with acquiring the image, thereby displaying the screen within at least a portion of the display area, and Based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera (310) is located, by rendering the image using a portion of the depth values, display the screen outside the at least a portion of the display area. causing the above wearable device (100), Wearable device (100).

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, Based on the event for displaying the screen within at least a portion of the display area where the gaze of the user (120) identified through the at least one second camera (310) is located, by rendering the image by applying a first depth value (550-1) for a first visual object (240) included in the image to the first visual object (240) and applying a second depth value (550-2) for a second visual object (250) included in the image to the second visual object (250), and displaying the screen within at least a portion of the display area, and Based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera (310) is located, by rendering the image by applying the first depth value (550-1) of the first depth value (550-1) for the first visual object (240) and the second depth value (550-2) for the second visual object (250) to the first visual object (240) and the second visual object (250), so as to display the screen outside the at least a portion of the display area. causing the above wearable device (100), Wearable device (100).

3. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, Based on the event for displaying the screen within the at least a portion of the display area where the gaze of the user (120) identified through the at least one second camera (310) is located, further using a denoising filter configured to remove noise from the screen and a sharpening filter configured to improve the sharpness of the screen to render the image, thereby displaying the screen within the at least a portion of the display area, and Based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera (310) is located, rendering the image by further using another denoising filter configured to perform less denoising than the denoising filter and another sharpening filter configured to perform less sharpening than the sharpening filter, so as to display the screen outside the at least a portion of the display area. causing the above wearable device (100), Wearable device (100).

4. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, Based on the event that the distance between the user (120) of the wearable device (100) and the boundary of the recommended area (610) recommended for the movement of the user (120) is less than a threshold distance, the image is rendered using the depth values ​​acquired in connection with acquiring the image, so as to display the screen within at least a part of the display area. causing the above wearable device (100), Wearable device (100).

5. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, While the content (220) is displayed outside at least a portion of the display area where the gaze identified through the at least one second camera (310) is located, based on the event for displaying the screen outside at least a portion of the display area, rendering the image using the portion of the depth values, so as to display the screen outside at least a portion of the display area, causing the above wearable device (100), Wearable device (100).

6. In claim 1, the wearable device, Further comprising a sensor configured to detect movement of the wearable device (100), The above instructions, when individually or collectively executed by the at least one processor, Through the above sensor, movement data of the wearable device (100) is acquired, and To identify the event for displaying the screen generated using the movement data and the image acquired through the at least one first camera (309), causing the above wearable device (100), Wearable device (100).

7. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, Based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera (310) is located, to reduce the FPS (frames per second) of the at least one first camera (309) to reduce the power consumed by the at least one first camera (309), causing the above wearable device (100), Wearable device (100).

8. A method executed in a wearable device (100) having a display assembly (308) including at least one first camera (309), at least one second camera (310), and at least one display including a display area, An operation for identifying an event for displaying a screen generated using an image acquired through at least one first camera (309); An operation of displaying the screen within at least a portion of the display area, based on the event for displaying the screen within at least a portion of the display area where the gaze of the user (120) identified through at least one second camera (310) is located, by rendering the image using depth values ​​acquired in connection with acquiring the image, and An operation of displaying the screen outside the at least part of the display area by rendering the image using a part of the depth values ​​based on the event for displaying the screen outside the at least part of the display area where the gaze identified through the at least one second camera (310) is located, method.

9. In claim 8, An operation of displaying the screen within at least a portion of the display area based on the event for displaying the screen within at least a portion of the display area where the gaze of the user (120) identified through the at least one second camera (310) is located, by rendering the image by applying a first depth value (550-1) for a first visual object (240) included in the image to the first visual object (240) and applying a second depth value (550-2) for a second visual object (250) included in the image to the second visual object (250), and An operation of displaying the screen outside the at least a portion of the display area by applying the first depth value (550-1) of the first depth value (550-1) for the first visual object (240) and the second depth value (550-2) for the second visual object (250) to the first visual object (240) and the second visual object (250), based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera (310) is located, to render the image, comprising: method.

10. In claim 8, An operation of displaying the screen within at least a portion of the display area based on the event for displaying the screen within at least a portion of the display area where the gaze of the user (120) identified through at least one second camera (310) is located, by further using a denoising filter configured to remove noise from the screen and a sharpening filter configured to improve the sharpness of the screen to render the image, and An operation of displaying the screen outside the at least a portion of the display area, based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera (310) is located, by further rendering the image using another denoising filter configured to perform less denoising than the denoising filter and another sharpening filter configured to perform less sharpening than the sharpening filter, method.

11. In claim 8, An operation of displaying the screen within at least a portion of the display area by rendering the image using the depth values ​​acquired in connection with acquiring the image based on the event that the distance between the user (120) of the wearable device (100) and the boundary of the recommended area (610) recommended for the movement of the user (120) is less than a threshold distance, method.

12. In claim 8, An operation of displaying the screen outside the at least part of the display area, based on the event for displaying the screen outside the at least part of the display area, by rendering the image using the part of the depth values, while the content (220) is displayed outside the at least part of the display area where the gaze identified through the at least one second camera (310) is located, method.

13. In claim 8, the wearable device (100) Further comprising a sensor configured to detect movement of the wearable device (100), An operation of obtaining movement data of the wearable device (100) through the above sensor, and An operation for identifying the event for displaying the screen generated using the movement data and the image acquired through the at least one first camera (309), method.

14. In claim 8, An operation of reducing the FPS (frames per second) of the at least one first camera (309) to reduce power consumed by the at least one first camera (309) based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera (310) is located. method.

15. In a non-transitory computer-readable storage medium storing one or more programs, the one or more programs are: When executed by a wearable device (100) having a display assembly (308) including at least one first camera (309), at least one second camera (310), and at least one display including a display area, Identify an event for displaying a screen generated using an image acquired through at least one first camera (309), Based on the event for displaying the screen within at least a portion of the display area where the gaze of the user (120) identified through the at least one second camera (310) is located, the image is rendered using depth values ​​acquired in connection with acquiring the image, thereby displaying the screen within at least a portion of the display area, and Based on the event for displaying the screen outside the at least a portion of the display area where the gaze identified through the at least one second camera (310) is located, by rendering the image using a portion of the depth values, display the screen outside the at least a portion of the display area. Including instructions that cause the above wearable device (100), Non-transitory computer-readable storage medium.

Images