Wearable device for changing frame rate related to foveated rendering according to moving speed of gaze position, and method therefor

Abstract

A wearable device according to one embodiment can acquire information about a gaze position by using at least one sensor. The wearable device can: acquire, on the basis of the identifying of the movement speed of the gaze position, which is slower than a reference speed, a plurality of first images corresponding to a display area of a display system by using the information; acquire, on the basis of the gaze position, a second image corresponding to a foveated portion specified in the display area; and perform foveated rendering on a screen to be displayed in the display area by combining the second image with each of the plurality of first images upscaled on the basis of the size of the display area.

Description

Wearable device and method for changing frame rate related to foveated rendering according to movement speed of gaze position

[0001] The disclosure relates to a wearable device and method for changing a frame rate associated with foveated rendering depending on a movement speed of a gaze position.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] The above information is provided solely as background information to assist in understanding the disclosure. No determination or assertion is made as to whether any of the above applies as background technology related to the disclosure.

[0004] Aspects of the disclosure are intended to at least address the problems and / or drawbacks described above, and to at least provide the advantages described below. Accordingly, one aspect of the disclosure is to provide a wearable device and method for changing the frame rate associated with foveated rendering based on the movement speed of the gaze position.

[0005] Additional aspects will be described in part by the description which follows, and in part will become apparent from the description, or may be learned by practice of the disclosed embodiments.

[0006] According to one embodiment of the disclosure, a wearable device is provided. The wearable device includes a display system including a first display and a second display, each of which is configured to be positioned toward the eyes of a user wearing the wearable device, respectively, at least one sensor, a memory storing one or more computer programs, and one or more processors communicatively coupled to the display, the at least one sensor, and the memory. The computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the wearable device to obtain information about a gaze position using the at least one sensor. The computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the wearable device to obtain a plurality of first images corresponding to a display area of ​​the display system based on identifying, using the information, a movement speed of the gaze position that is slower than a reference speed. The computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the wearable device to acquire a second image corresponding to a foveated portion specified within the display area based on the gaze position.The computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the wearable device to perform foveated rendering on a screen to be displayed through the display area by combining the second image with each of the plurality of first images upscaled based on a size of the display area.

[0007] According to another aspect of the disclosure, a method of a wearable device is provided, the method including a display system including a first display and a second display, the first display and the second display being arranged to face each of the eyes of a user wearing the wearable device, and at least one sensor. The method may include an operation of obtaining information about a gaze position using the at least one sensor. The method may include an operation of obtaining a plurality of first images corresponding to a display area of ​​the display system based on identifying a movement speed of the gaze position that is slower than a reference speed using the information. The method may include an operation of obtaining a second image corresponding to a foveated portion specified within the display area based on the gaze position. The method may include an operation of performing foveated rendering on a screen to be displayed through the display area by combining the second image with each of the plurality of first images, which are upscaled based on a size of the display area.

[0008] In one embodiment, a wearable device may include at least one display, at least one sensor, at least one processor including processing circuitry, and a memory including one or more storage media storing instructions. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display an image on an entire display area of ​​the at least one display at a first frame rate while identifying a movement speed of a gaze position higher than a first reference speed through the at least one sensor. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display an image on a foveated portion of the display area at a second frame rate lower than the first frame rate, and to display an image on a peripheral portion of the display area while identifying a movement speed of a gaze position lower than the first reference speed and higher than a second reference speed through the at least one sensor. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display an image in the foveated portion at a third frame rate lower than the second frame rate, and to display an image in the peripheral portion at the second frame rate, while identifying a movement speed of the gaze position lower than the second reference speed through the at least one sensor.

[0009] In one embodiment, a method of a wearable device including at least one display and at least one sensor may be provided. The method may include displaying an image on an entire display area of ​​the at least one display at a first frame rate while identifying a movement speed of a gaze position higher than a first reference speed through the at least one sensor. The method may include displaying an image on a foveated portion of the display area at a second frame rate lower than the first frame rate, and displaying the image on a peripheral portion of the display area, while identifying a movement speed of a gaze position lower than the first reference speed and higher than a second reference speed through the at least one sensor. The method may include displaying an image on the foveated portion at a third frame rate lower than the second frame rate, and displaying the image on the peripheral portion at the second frame rate, while identifying a movement speed of a gaze position lower than the second reference speed through the at least one sensor.

[0010] In one embodiment, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when executed by a wearable device comprising a display system including a first display and a second display, the first display and the second display being configured to be positioned toward respective eyes of a user wearing the wearable device, and at least one sensor, may cause the wearable device to control the display system such that images acquired at a first frame rate are displayed in a foveated portion and a peripheral portion, respectively, while the wearable device identifies a movement speed of the gaze position that is higher than a reference speed through the at least one sensor. The instructions, when executed by the wearable device, may cause the wearable device to control the display system such that images acquired at the first frame rate are displayed in the peripheral portion and images acquired at a second frame rate that is lower than the first frame rate are displayed in the foveated portion, while the wearable device identifies a movement speed of the gaze position that is lower than the reference speed through the at least one sensor.

[0011] In one embodiment, a wearable device may include a display system including a first display and a second display, each of which is configured to be positioned toward each eye of a user wearing the wearable device, at least one sensor, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to control the display system such that images acquired according to a first frame rate are displayed in each of a foveated portion and a peripheral portion while identifying a movement speed of a gaze position higher than a reference speed through the at least one sensor. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to control the display system such that an image acquired according to the first frame rate is displayed in the peripheral portion and an image acquired according to a second frame rate lower than the first frame rate is displayed in the foveated portion while identifying a movement speed of the gaze position lower than the reference speed through the at least one sensor.

[0012] Other aspects, advantages, and key features of the disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments of the disclosure, taken in conjunction with the accompanying drawings.

[0013] The above and other aspects, features, and advantages of some embodiments of the disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0014] FIG. 1 illustrates an embodiment of a wearable device performing foveated rendering according to one embodiment of the disclosure;

[0015] FIGS. 2A and 2B illustrate block diagrams of a wearable device according to various embodiments of the disclosure;

[0016] FIG. 3A and FIG. 3B illustrate a flowchart of a wearable device according to various embodiments of the disclosure;

[0017] FIG. 4 illustrates the operation of a wearable device for determining information about gaze position, according to one embodiment of the disclosure;

[0018] FIG. 5 illustrates the operation of a wearable device for determining a frame rate based on a movement speed of a gaze position, according to one embodiment of the disclosure;

[0019] FIGS. 6A and 6B illustrate flowcharts of operations of a wearable device related to foveated rendering according to various embodiments of the disclosure;

[0020] FIGS. 7A and 7B illustrate flowcharts of operations of a wearable device for detecting a gaze position moving at a speed within the first speed range of FIG. 5 according to various embodiments of the disclosure;

[0021] FIGS. 8A and 8B illustrate flowcharts of operations of a wearable device for detecting a gaze position moving at a speed within the third speed range of FIG. 5 according to various embodiments of the disclosure;

[0022] FIGS. 9A and 9B illustrate flowcharts of operations of a wearable device for detecting a gaze position moving at a speed within the second speed range of FIG. 5 according to various embodiments of the disclosure; and

[0023] FIGS. 10A and 10B illustrate the appearance of a wearable device according to various embodiments of the disclosure.

[0024] It should be noted that throughout the drawings, similar drawing symbols are used to describe the same or similar elements, features, and structures.

[0025] The following description, with reference to the accompanying drawings, is provided to facilitate a comprehensive understanding of various embodiments of the disclosure as defined by the claims and their equivalents. While it includes numerous specific details to aid understanding, these are intended to be illustrative only. Accordingly, those skilled in the art will recognize that various modifications and variations of the various embodiments disclosed herein may be made without departing from the scope of the disclosure. Additionally, well-known features and structures may be omitted for clarity and brevity.

[0026] The terms and words used in the following description and claims are not limited to their bibliographic meanings. Therefore, it will be apparent to those skilled in the art that the following descriptions of various embodiments of the disclosure are provided for illustrative purposes only and are not intended to limit the disclosure to the scope defined by the appended claims and their equivalents.

[0027] The singular forms "a," "an," and "the" should be understood to include plural references unless the context explicitly dictates otherwise. Thus, "a component surface" includes reference to one or more such surfaces.

[0028] In the document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" can include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" can modify the components without regard to order or importance, and are only used to distinguish one component from another, but do not limit the components. When it is said that a component (e.g., a first component) is "(functionally or communicatively) connected" or "connected" to another component (e.g., a second component), the component can be directly connected to the other component, or can be connected via another component (e.g., a third component).

[0029] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0030] It is apparent that each block of the flowchart, and combinations of flowcharts, can be performed by one or more computer programs comprising computer-executable instructions. The entirety of the one or more computer programs may be stored in a single memory device, or the one or more computer programs may be divided into different portions stored in different memory devices.

[0031] Any of the functions or operations disclosed herein may be processed by a single processor or a combination of processors. A single processor or a combination of processors may include a circuit that performs processing, and / or an application processor (AP) (e.g., a central processing unit (CPU)), a communication processor (CP) (e.g., a modem), a graphical processing unit (GPU), a neural processing unit (NPU)9 (e.g., an artificial intelligence (AI) chip), a wireless-fidelity (Wi-Fi) chip, or a Bluetooth TM A circuit including a chip, a global positioning system (GPS) chip, a near field communication (NFC) chip, a connectivity chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio CODEC chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on chip (SoC), an IC, or the like.

[0032] FIG. 1 illustrates an embodiment of a wearable device performing foveated rendering according to one embodiment of the disclosure.

[0033] Referring to FIG. 1, the wearable device (101) may include a head-mounted display (HMD) that can be worn on a user's head. The wearable device (101) may be referred to as a head-mounted display (HMD) device, a headgear electronic device, a glasses-type (or goggle-type) electronic device, a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device.

[0034] A wearable device (101) designed to block external light directed to a user's eyes while worn on a user's head is illustrated, but the embodiment is not limited thereto. An example of a hardware configuration included in the wearable device (101) is exemplarily described with reference to FIG. 2A. An example of a structure of a wearable device (101) wearable on a user's head is described with reference to FIG. 10A and / or FIG. 10B. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may include an accessory (e.g., a strap) for attaching to the user's head.

[0035] According to one embodiment of the disclosure, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, while a user wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Since the wearable device (101) combines the ambient light and the light emitted from the display, the user may see a mixed image of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0036] According to one embodiment of the disclosure, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). For example, in a state where a user wears the wearable device (101), the wearable device (101) may include a housing that covers the user's eyes. In the state, the wearable device (101) may include a display arranged on a first surface of the housing facing the eyes. Referring to FIG. 1, the wearable device (101) may include a first display (110-1) configured to face the left eye (140-1) of a user wearing the wearable device (101) and a second display (110-2) configured to face the right eye (140-2) of a user wearing the wearable device (101).

[0037] According to one embodiment of the disclosure, a wearable device (101) may include at least one image sensor configured to be arranged toward a user's eyes. Referring to FIG. 1, the wearable device (101) may include a first eye tracking camera (ET CAM) (130-1) configured to face a left eye (140-1) of a user wearing the wearable device (101) and a second eye tracking camera (130-2) configured to face a right eye (140-2) of a user wearing the wearable device (101). The wearable device (101) may obtain images and / or videos of the user's two eyes (e.g., the left eye (140-1) and / or the right eye (140-2)) from the first eye tracking camera (130-1) and / or the second eye tracking camera (130-2), and may calculate or determine a location gazed by the user. The location detected by the wearable device (101) and gazed at by the user may be referred to as a gaze location (g), gaze point, and / or gaze point.

[0038] Referring to FIG. 1, a screen (120) displayed by a wearable device (101) is illustrated. The wearable device (101) can display the screen (120), which includes content having binocular disparity (b), on a first display (110-1) and a second display (110-2). For example, the wearable device (101) can change a first position of the content within the first display (110-1) and a second position of the content within the second display (110-2), thereby changing the distance of the content from the wearable device (101) perceived by the user. For example, by adjusting the difference between the first position and the second position (e.g., the difference in position on the horizontal axis of the screen (120)), the wearable device (101) can change the perspective of the content. As the above difference increases, the binocular disparity (b) formed between the two eyes when viewing the content may increase, and the wearable device (101) may provide the user viewing the content with a sensation that the content is approaching. As the above difference decreases, the binocular disparity (b) of the content may decrease, and the wearable device (101) may provide the user viewing the content with a sensation that the content is moving away.

[0039] According to one embodiment of the disclosure, a display area formed by at least one display (e.g., a first display (110-1) and / or a second display (110-2)) within a wearable device (101) may cover the entire viewable area of ​​a user while the wearable device (101) is worn by the user. To provide an immersive user experience (e.g., mixed reality and / or virtual reality based on video see through (VST)), the wearable device (101) may display a high-resolution screen (120). To display a high-resolution screen (120), at least one display included in the wearable device (101) may include a large number of pixels. In the disclosure, the term “resolution” is used to refer to the density of pixels of an image and / or a display. The density and / or resolution of pixels may be measured or parameterized based on units of pixels per inch (ppi) and / or dots per inch (dpi). For example, a lower resolution of a first image than a second image may indicate that the density of pixels in the first image is lower than the density of pixels in the second image.

[0040] According to one embodiment of the disclosure, the wearable device (101) can visualize the screen (120) using one or more images that are smaller than the size of the display area and / or have a lower resolution than the display area. For example, the wearable device (101) can enlarge (e.g., upscale) a first image (151) that is smaller than the size of the display area and / or has a lower resolution than the display area to fill the entire display area. Referring to FIG. 1, a display area having a size of a width (wd) and a height (hd) and a first image (151) having a width (w1) less than the width (wd) and a height (h1) less than the height (hd) are exemplarily illustrated. The wearable device (101) can acquire or generate an enlarged first image (151) corresponding to the entire display area by enlarging the first image (151) according to a ratio between the size of the display area and the size of the first image (151) (e.g., wd / w1 and / or hd / h1).

[0041] In one embodiment of the disclosure, a first image (151) having a lower resolution than the resolution of the entire display area may be provided from a software application executed by the wearable device (101), or may be generated by the software application. Since the software application is executed using a resolution lower than the resolution of the entire display area, the wearable device (101) can acquire the first image (151) to be used for displaying the screen (120) using fewer resources (e.g., processor occupancy, processor computational load, battery power consumption, bandwidth between the processor and memory, and / or memory capacity) than are required to acquire another image having the resolution of the entire display area.

[0042] In one embodiment of the disclosure, the wearable device (101) can perform foveated rendering. A user looking in a specific direction clearly perceives a portion of the field of view that includes the specific direction, and perceives the remainder of the field of view unclearly. The expression "foveated rendering" may be used to refer to an operation of the wearable device (101) to visualize a portion (129) of the display area that at least partially overlaps with the gaze position (g) of a user wearing the wearable device (101) more clearly than the remainder of the display area. The portion (129) may be referred to as the foveated portion, and the remainder of the display area that is distinct from the portion (129) may be referred to as the peripheral portion. For example, foveated rendering may be performed to intensively visualize a portion (129) that is relatively clearly perceived, so that the resources of the wearable device (101) may be utilized efficiently (or optimally). The operation of a wearable device (101) performing foveated rendering is described with reference to FIG. 3a and / or FIG. 3b.

[0043] Referring to FIG. 1, images (e.g., a first image (151) and a second image (152)) used for foveated rendering are illustrated. For example, the wearable device (101) may acquire the first image (151) and / or the second image (152) having a size less than the size of the entire display area based on the execution of a software application. For example, the sizes of the first image (151) and the second image (152) may be smaller than the size of the entire display area. The first image (151) and the second image (152) may have different resolutions. For example, the resolution of the first image (151) may be lower than the resolution of the entire display area. For example, the resolution of the second image (152) may be equal to or less than the resolution of the entire display area. For example, the resolution of the second image (152) may be higher than the resolution of the first image (151). In one embodiment of executing a software application to acquire a first image (151) and a second image (152) that are synchronized with each other, the wearable device (101) can acquire a second image (152) corresponding to a portion (159) of the first image (151). For example, the wearable device (101) can acquire a second image (152) that more clearly expresses the content of the portion (159) of the first image (151).

[0044] In one embodiment of the disclosure, the wearable device (101) can enlarge the first image (151) using the size and / or resolution of the entire display area to fill the entire display area. For example, the first image (151) can be associated with the entire display area. In one embodiment of obtaining the first image (151) associated with the entire display area, the wearable device (101) can determine or detect a portion (159) of the first image (151) corresponding to a portion (129) of the display area that at least partially overlaps the gaze position (g). The wearable device (101) can calculate or detect the gaze position (g) used to determine the portion (159) using at least one sensor (e.g., the first gaze tracking camera (130-1) and / or the second gaze tracking camera (130-2)) for detecting the gaze position (g).

[0045] In one embodiment of the disclosure, the wearable device (101) can detect or calculate a gaze position (g) within a display area using information about at least one eye (e.g., a left eye (140-1) and / or a right eye (140-2)) of a user wearing the wearable device (101) and / or a motion of the wearable device (101) attached to the head of the user (or a motion of the head). An operation of the wearable device (101) determining a gaze position (g) within the display area and / or a portion (129) of the display area that at least partially overlaps the gaze position (g) according to one embodiment of the disclosure is described with reference to FIG. 4.

[0046] Referring to FIG. 1, a portion (159) of a first image (151) may correspond to a portion (129) of a display area overlapping a gaze position (g). The wearable device (101) may acquire or generate a second image (152) corresponding to the portion (159) of the first image (151) and having a second resolution exceeding the first resolution of the first image (151). The second image (152) may be referred to as a partial image of the first image (151) from the viewpoint of corresponding to the portion of the first image (151), and the first image (151) may be referred to as a full image. Depending on the difference in resolution, the first image (151) may be referred to as a low-resolution image, and the second image (152) may be referred to as a high-resolution image. As a non-limiting example, the width (w1) and height (h1) of the first image (151) may be equal to the width (w2) and height (h2) of the second image (152). The sizes of the first image (151) and the second image (152) may all be smaller than the size of the entire display area (or screen (120)). For example, the size of the second image (152) may substantially match the size of a portion (129) of the display area corresponding to a portion (159) of the first image (151).

[0047] According to one embodiment of the disclosure, the wearable device (101) may control at least one display to display a first image (151) mapped to the entire display area and a second image (152) according to a location of a portion (159) within the first image (151). For example, within a screen (120) displayed in the entire display area, the first image (151) may be enlarged according to the size of the entire display area, and the second image (152) may be positioned in a portion (129) of the display area related to the portion (159). For example, since the first image (151), which has a lower resolution than the resolution of the display, is mapped to the entire display area, the resolution of the first image (151), which is enlarged to be mapped to the entire display area, may be lower than the resolution of the display. For example, when a second image (152) having the resolution of the display and the size of the portion (129) is displayed on the portion (129), the resolution of the second image (152) displayed through the portion (129) may be substantially the same as the resolution of the display.

[0048] Referring to FIG. 1, in one embodiment of displaying a second image (152) in a portion (129) overlapping with a gaze position (g), the wearable device (101) can clearly provide content within the portion (129) to a user gazing at the portion (129) by using the second image (152) having a higher resolution than the resolution of the first image (151). According to one embodiment of the disclosure, the wearable device (101) can measure or calculate a displacement (e.g., velocity and / or acceleration) of the gaze position (g) within the display area based on the motion of at least one eye of the user (e.g., the left eye (140-1) or the right eye (140-2)). For example, the wearable device (101) can obtain or determine information about the gaze position (g) by using at least one sensor.

[0049] In one embodiment of the disclosure, the wearable device (101) can change or determine parameters related to foveated rendering (e.g., the size of the foveated portion, the period for performing foveated rendering, the frequency, and / or the frame rate) based on information about the gaze position (g). For example, the wearable device (101) can determine whether to acquire a second image (152) using a software application executed to generate the screen (120) based on information about the gaze position (g). For example, the wearable device (101) can determine or calculate a period (e.g., a frame rate) for acquiring the second image (152) using the software application based on information about the gaze position (g). The operation of the wearable device (101) using the above information to control the execution of foveated rendering is described with reference to FIGS. 5, 6a, 6b, 7a, 7b, 8a, 8b, 9a, and 9b.

[0050] As described above, according to one embodiment of the disclosure, the wearable device (101) can perform dynamic foveated rendering to reduce or optimize the resources occupied for the foveated rendering. For example, the wearable device (101) can acquire or generate a screen (120) to be displayed over the entire display area using only the first image (151) without the second image (152). For example, the wearable device (101) can reduce the frequency (e.g., frame rate) of acquiring the second image (152), which is to be combined onto the first image (151) enlarged for display within the entire display area, to a frequency lower than the frequency of acquiring the first image (151). For example, a user moving the gaze position (g) quickly may mean that the user is not focusing on a portion (129) of the display area that overlaps the gaze position (g). The wearable device (101) can estimate the user's intention using the movement speed of the gaze position (g). Using the estimated intention, the wearable device (101) can reduce the frequency of acquiring the second image (152) to provide clear content. For example, depending on the movement speed of the gaze position (g), the wearable device (101) can increase (e.g., while the movement speed decreases) or decrease (e.g., while the movement speed increases) the frequency of performing foveated rendering using the second image (152).

[0051] Hereinafter, the hardware configuration of a wearable device (101) configured to perform foveated rendering is described with reference to FIG. 2a and / or FIG. 2b.

[0052] FIGS. 2A and 2B illustrate block diagrams of a wearable device according to various embodiments of the disclosure. The wearable device (101) of FIGS. 2A and / or 2B may include the wearable device (101) of FIG. 1 .

[0053] Referring to FIG. 2A, a wearable device (101) according to one embodiment may include a processor (210), a memory (215), a display (110) (e.g., the first display (110-1) and / or the second display (110-2) of FIG. 1), and / or a sensor (220) (e.g., the image sensor (130) and / or the motion sensor (222)). The processor (210), the memory (215), the display (110), and / or the sensor (220) may be electrically and / or operatively connected to each other by electronic components, such as a communication bus (202). In the disclosure, the operative connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components, such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (101) is not limited to those illustrated in FIG. 2A. For example, the wearable device (101) may include only some of the electronic components illustrated in FIG. 2A.

[0054] According to one embodiment, a processor (210) of a wearable device (101) may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment of the disclosure, the wearable device (101) may include one or more processors. According to one embodiment of the disclosure, the structure of the processor (210) is not limited to one embodiment of the disclosure, and at least one circuit may be formed as a separate processor that is physically separated from the processor. The processor (210) may have a structure of a multi-core processor, such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor architecture of the processor (210) may include a architecture based on multiple core circuits (e.g., a big-little architecture) that are distinguished by power consumption, clock frequency, and / or computational amount per unit time. In one embodiment including the processor (210) having a multi-core processor architecture, the operations and / or functions of the disclosure may be performed individually or collectively by one or more cores included in the processor (210).

[0055] The memory (215) of the wearable device (101) according to one embodiment may include electronic components for storing data and / or instructions input to and / or output from the processor (210). The memory (215) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC). In one embodiment of the disclosure, the memory (215) may be referred to as storage.

[0056] In one embodiment of the disclosure, a display (110) of a wearable device (101) can output visualized information to a user of the wearable device (101). The display (110), which is arranged in front of the eyes of a user wearing the wearable device (101), can be arranged on at least a portion of a housing of the wearable device (101) (e.g., the first display (110-1) and / or the second display (110-2) of FIG. 1 ). For example, the display (110) can be controlled by a processor (210) including circuits such as a central processing unit (CPU) (211), a graphics processing unit (GPU) (212), and / or a display processing unit (DPU) (213), to output visualized information to the user. The display (110) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (110) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). The embodiment is not limited thereto, and for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (110) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment of the disclosure, the display (110) may be referred to as a display panel and / or a display module. The pixels included in the display (110) may be arranged to face either of the user's eyes when the wearable device (101) is worn by the user.For example, the display (110) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0057] In one embodiment of the disclosure, the sensor (220) of the wearable device (101) may generate electrical information that may be processed by the processor (210) and / or the memory (215) from non-electronic information related to the wearable device (101). For example, the sensor (220) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (101). In addition to the GPS method, the sensor (220) may generate information indicating the geographic location of the wearable device (101) based on a global navigation satellite system (GNSS) such as, for example, Galileo or Beidou (compass). The above information may be stored in memory (215), processed by a processor (210), and / or transmitted to another electronic device distinct from the wearable device (101) via a communication circuit.

[0058] Referring to FIG. 2A, an image sensor (130) and / or a motion sensor (222) are illustrated as examples of a sensor (220) included in a wearable device (101). The image sensor (130) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The image sensor (130) may be referred to as a camera. A plurality of optical sensors included in the image sensor (130) may be arranged in the form of a two-dimensional (2D) array. The image sensor (130) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and may generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the image sensor (130) may mean one (a) two-dimensional frame data acquired from the image sensor (130). For example, video data captured using the image sensor (130) may mean a sequence of a plurality of two-dimensional frame data acquired from the image sensor (130) according to a frame rate. The image sensor (130) may further include a flash light that is positioned toward the direction in which the image sensor (130) receives light and outputs light toward the direction.

[0059] According to one embodiment of the disclosure, a wearable device (101) may include, as an example, a plurality of image sensors arranged facing different directions, as an image sensor (130). As described above with reference to FIG. 1, the plurality of image sensors may include gaze tracking cameras (e.g., the first gaze tracking camera (130-1) and / or the second gaze tracking camera (130-2) of FIG. 1) configured to be arranged toward the eyes of a user wearing the wearable device (101). The plurality of image sensors may include outward cameras. The processor (210) may identify the direction of the user's gaze using images and / or videos acquired from the gaze tracking cameras. The gaze tracking cameras may include infrared (IR) sensors. The gaze tracking cameras may be referred to as eye sensors and / or eye trackers.

[0060] For example, the external camera may be positioned facing the front of a user wearing the wearable device (101) (e.g., in a direction that both eyes may face). The wearable device (101) may include multiple external cameras. The embodiment is not limited thereto, and the external camera may be positioned facing an external space. Using images and / or videos acquired from the external cameras, the processor (210) may identify external objects. For example, based on images and / or videos acquired from the external cameras, the processor (210) may identify the position, shape, and / or gesture (e.g., hand gesture) of a hand of a user wearing the wearable device (101). Using images and / or videos of the external environment acquired from the external cameras, the processor (210) may recognize or track one or more objects within the external environment.

[0061] According to one embodiment of the disclosure, the motion sensor (222) may output electrical signals representing gravitational accelerations, accelerations, and / or angular velocities of a plurality of axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other and based on a designated origin within the wearable device (101) and / or the motion sensor (222). For example, the processor (210) may repeatedly receive or acquire sensor data including accelerations, angular velocities, and / or magnitudes of magnetic fields of a number of the plurality of axes from the motion sensor (222) based on a designated period (e.g., 1 millisecond). In one embodiment of the disclosure, the motion sensor (222) may be referred to as an inertial measurement unit (IMU). The sensor (220) included in the wearable device (101) is not limited to those described above, and may include a grip sensor, a proximity sensor, a heart rate sensor, a fingerprint sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor. Using the motion sensor (222), the processor (210) can detect motion of the wearable device (101) (e.g., motion of the wearable device (101) caused by a user wearing the wearable device (101).

[0062] According to one embodiment of the disclosure, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (210) of the wearable device (101) may be stored in the memory (215) of the wearable device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (101) and / or the processor (210) may perform at least one of the operations of FIGS. 3A, 3B, 6A, 6B, 7A, 7B, 8A, 8B, 9A, and 9B when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, program, and / or software application is executed. Hereinafter, the fact that a software application is installed in a wearable device (101) may mean that one or more instructions provided in the form of a software application (or package) are stored in a memory (215), and that the one or more applications are stored in a format executable by the processor (210) (e.g., a file having an extension specified by the operating system of the wearable device (101)). As an example, the application may include a program and / or a library related to a service provided to a user.

[0063] Referring to FIG. 2A, programs installed in the wearable device (101) may be included in any one of different layers, including the application layer (240), the framework layer (250), and / or the hardware abstraction layer (HAL) (280), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (110), and / or the sensor (220)) of the wearable device (101) may be included in the hardware abstraction layer (280). The framework layer (250) may be referred to as an XR framework layer from the perspective of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 2A may be logically (or for convenience of explanation) separated, and may not mean that the address space of the memory (215) is separated by the layers.

[0064] For example, within the framework layer (250), programs designed to target at least one of the hardware abstraction layer (280) and / or the application layer (240) (e.g., a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye-gaze tracker (274), and / or a face tracker (275)) may be included. The programs included in the framework layer (250) may provide an application programming interface (API) that is executable (or callable) based on other programs.

[0065] For example, the application layer (240) may include a program designed to target users of the wearable device (101). As an example of programs included in the application layer (240), an extended reality (XR) system user interface (UI) (241) and / or an XR application (242) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (240) may call an API to cause execution of functions supported by programs included in the framework layer (250).

[0066] For example, the wearable device (101) may display one or more visual objects on the display (110) for performing interaction with the user based on the execution of the XR system UI (241). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user with functions available within a virtual space based on the execution of the XR system UI (241).

[0067] Referring to FIG. 2A, a lightweight renderer (243) and / or an XR plug-in (244) are illustrated to be included within the XR system UI (241), but are not limited thereto. For example, based on the XR system UI (241), the processor (210) may execute a lightweight renderer (243) and / or an XR plug-in (244) within the framework layer (250).

[0068] For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (243). The lightweight renderer (243) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (243) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (244). The XR plugin (244) can be referred to as an open XR native client from the perspective of defining (or setting up) the entire rendering pipeline.

[0069] For example, the wearable device (101) may display a screen representing at least a portion of a virtual space on the display (110) based on the execution of the XR application (242). The XR plug-in (244-1) included in the XR application (242) may include instructions that support functions similar to those of the XR plug-in (244) of the XR system UI (241). Descriptions of the XR plug-in (244-1) that overlap with those of the XR plug-in (244) may be omitted. The wearable device (101) may cause the execution of the virtual space manager (251) based on the execution of the XR application (242).

[0070] According to one embodiment of the disclosure, a wearable device (101) may provide a virtual space service based on the execution of a virtual space manager (251). For example, the virtual space manager (251) may include a platform for supporting a virtual space service. Based on the execution of the virtual space manager (251), the wearable device (101) may identify a virtual space formed based on a user's location indicated by data acquired through a sensor (220), and may display at least a portion of the virtual space on a display (110). The virtual space manager (251) may be referred to as a composition presentation manager (CPM).

[0071] For example, the virtual space manager (251) may include a runtime service (252). As an example, the runtime service (252) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (252). As an example, the wearable device (101) may perform rendering for a virtual space service to the user based on the execution of the runtime service (252). For example, a function related to a virtual space, executable by the application layer (240), may be supported based on the execution of the runtime service (252).

[0072] For example, the virtual space manager (251) may include a pass-through manager (253). Based on the execution of the pass-through manager (253), the wearable device (101) may display a screen representing a virtual space (e.g., screen (120) of FIG. 1) on the display (110), while displaying an image and / or video representing an actual space acquired through an external camera by overlaying it on at least a portion of the screen.

[0073] For example, the virtual space manager (251) may include an input manager (254). The wearable device (101) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (270) based on the execution of the input manager (254). The wearable device (101) may use the acquired data to identify user input related to the wearable device (101). The user input may be related to a motion (e.g., a hand gesture), gaze, and / or speech of the user identified by a sensor (220) (e.g., an image sensor (130) such as an external camera). The user input may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0074] For example, the perception abstract layer (260) can be used for data exchange between the virtual space manager (251) and the perception service layer (270). From the perspective of being used for data exchange between the virtual space manager (251) and the perception service layer (270), the perception abstract layer (260) can be referred to as an interface. For example, the perception abstract layer (260) can be referenced as OpenPX. The perception abstract layer (260) can be used for a perception client and a perception service.

[0075] According to one embodiment of the disclosure, the recognition service layer (270) may include one or more programs for processing data acquired from the sensor (220). The one or more programs may include at least one of a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye tracker (274), and / or a face tracker (275). The type and / or number of the one or more programs included in the recognition service layer (270) are not limited to those illustrated in FIG. 2A.

[0076] For example, the wearable device (101) can identify the pose of the wearable device (101) using the sensor (220) based on the execution of the position tracker (271). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using an external camera (e.g., an image sensor (130)) and / or an IMU (e.g., a motion sensor (222) including a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (271). The position tracker (271) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0077] For example, the wearable device (101) may obtain information for providing a three-dimensional (3D) virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of the space recognizer (272). The wearable device (101) may reproduce the surrounding environment of the wearable device (101) in three dimensions using data obtained using an external camera (e.g., an image sensor (130)) based on the execution of the space recognizer (272). The wearable device (101) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (101) reproduced in three dimensions based on the execution of the space recognizer (272). The space recognizer (272) may be referred to as a scene understanding (SU) module (or a scene recognition program).

[0078] For example, the wearable device (101) may identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (101) based on the execution of the gesture tracker (273). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user using data acquired from an external camera (e.g., an image sensor (130)) based on the execution of the gesture tracker (273). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using an external camera based on the execution of the gesture tracker (273). The gesture tracker (273) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0079] For example, the wearable device (101) may identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (274). As an example, the wearable device (101) may identify eye movements of the user using data acquired from a gaze tracking camera (e.g., an image sensor (130)) based on the execution of the gaze tracker (274). The gaze tracker (274) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0080] For example, the wearable device (101) may obtain or generate information related to the face of a user of the wearable device (101) based on the execution of the face tracker (275). For example, the wearable device (101) may obtain or generate information related to the movement and / or expression of the face of a user wearing the wearable device (101) from data obtained from the image sensor (130) based on the execution of the face tracker (275). The face tracker (275) may be referred to as a face tracking (FT) (or a face tracking program) and / or a face tracking module.

[0081] Referring to FIG. 2A, examples of a processor (210) include a CPU (211), a GPU (212), and / or a display processing unit (DPU) (213). A foveated renderer (290) may include instructions for foveated rendering. A processor (210) (e.g., a DPU (213)) executing the foveated renderer (290) may obtain at least one image to be at least partially displayed in a display area of ​​a display (110) from a software application (e.g., a software application executed by the CPU (211) and / or the GPU (212). The processor (210) executing the foveated renderer (290) can divide the display area of ​​the display (110) into a foveated portion (e.g., portion (129) of FIG. 1) and a peripheral portion using the gaze position calculated using the position tracker (271) and / or the gaze tracker (274). For example, the processor (210) detecting the coordinate values ​​of the gaze position can determine the portion of the display area including the coordinate values ​​as the foveated portion. The DPU (213) executing the foveated renderer (290) can obtain at least one image corresponding to each of the foveated portion and the peripheral portion, and having a size smaller than the size of the entire display area of ​​the display (110) or a resolution lower than the resolution of the display area.

[0082] The processor (210) executing the foveated renderer (290) can obtain or generate a composite image to be displayed on the display (110) by synthesizing an image corresponding to the foveated portion and an image corresponding to the peripheral portion. For example, the processor (210) can perform upscaling to enlarge the image corresponding to the peripheral portion to the size of the entire display area of ​​the display (110). On the enlarged image, the processor (210) can combine the image corresponding to the foveated portion to generate a composite image to be displayed on the display (110). Along the boundary line of the image corresponding to the foveated portion, the processor (210) can apply a visual effect, such as blur, to blend the enlarged image and the image corresponding to the foveated portion.

[0083] Referring to FIG. 2B, the wearable device (101) can execute a virtual space manager (251) (composition presentation manager, CPM), which is a program for rendering based on a virtual space. The virtual space manager (251) can include a platform for supporting a virtual space service. The virtual space manager (251) can include a runtime service (252) (e.g., OpenXR Runtime), a panel rendering (255) (e.g., 2D Panel Renderer), and an XR composition unit (256) (XR compositor). The wearable device (101) can execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (252). The wearable device (101) can display at least one image (video) on a panel (e.g., a 2D panel) to be displayed through a display based on the execution of the panel rendering (255). The wearable device (101) can synthesize an image of an external space (hereinafter, a pass-through image) captured by a camera in a virtual space and an image of a virtual area based on the execution of the XR synthesis unit (256). The wearable device (101) can execute the XR synthesis unit (256) to merge the pass-through image and the image of the virtual area to generate a composite image. The wearable device (101) can transmit the composite image to a display buffer so that the composite image is displayed.

[0084] The wearable device (101) can execute a space flinger (291). The space flinger (291) can be a program configured to support functions for displaying an image in a three-dimensional virtual space. The wearable device (101) can perform a preprocessing operation for rendering based on the virtual space manager (251) by executing the space flinger (291). For example, the wearable device (101) can process image information provided by an application (e.g., an XR application (242), applications (240) that provide a general 2D screen other than XR, and an XR system UI (241) provided by a system application) based on the execution of the space flinger (291). The space flinger (291) can include a system screen manager (292), an input router (293), and / or an impress engine (294).

[0085] By executing the system screen manager (292) included in the space flinger (291), the wearable device (101) can display the system UI (241). The system UI (241) can be loaded by the wearable device (101) executing the system screen manager (292) based on a call of a spatializer API and / or a same-process private API. The space flinger (291) can determine the layout (e.g., position and / or display order) of the XR system UI (241) within the screen. The system screen manager (292) can transmit information (e.g., image information) for rendering the XR system UI (241) according to the determined layout to the virtual space manager (251).

[0086] An input router (293) included in the space flinger (291) may be a program configured to process user input (e.g., user input on a system screen and / or an application screen). The input router (293) may map a user input detected by a sensor of the wearable device (101) to at least one of one or more software applications mapped to a virtual space by the space flinger (291) (e.g., an XR application (242), applications (240), and / or a system application executed to provide an XR system UI (241)). For example, mapping a user input may include executing instructions (e.g., a subroutine and / or an event handler) of a software application for processing the user input.

[0087] The impression engine (294) included in the space flinger (291) may be a renderer (e.g., a lightweight renderer) for generating an image to be positioned in the virtual space of the virtual space manager (251). The impression engine (294) may be executed for rendering for the XR system UI (241). If there are insufficient resources for rendering based on the impression engine (294), the wearable device (101) may execute an external rendering engine.

[0088] Referring to FIG. 2b, an XR application (242) is illustrated as an example of software applications executable by a wearable device (101). The XR application (242) may include an immersive application, such as a 3D game. The wearable device (101) executing the XR application (242) may execute a virtual space manager (251) to perform rendering of a virtual space based on the XR application (242).

[0089] Referring to FIG. 2B, examples of software applications executable by the wearable device (101) include an XR application (242) and different applications (240) (e.g., a first application (240-1), a second application (240-2), ..., an Nth application (240-N)). Based on the execution of the application (240), the wearable device (101) can obtain an image (e.g., a window and / or an activity) having a form of a two-dimensional panel (e.g., a square and / or a square with rounded corners). The wearable device (101) that executes the application (240) can execute a space flinger (291) to provide the image acquired based on the execution of the application (240) through a virtual space. A wearable device (101) executing a spatial flinger (291) can obtain information related to the image (e.g., RGB information based on an object referred to as SurfaceComposer) from an application (240). The wearable device (101) can obtain the information from the application (240) through a spatializer API. By executing the spatial flinger (291), the wearable device (101) can generate information (e.g., dual image information based on binocular disparity) to be used to display the image in three dimensions. The information can be processed by a virtual space manager (251).

[0090] The foveated rendering of the disclosure can be performed by the XR synthesis unit (256) of FIG. 2B. The wearable device (101) executing the XR synthesis unit (256) can obtain information for foveated rendering from the spatial flinger (291). The wearable device (101) that has obtained the information can generate or obtain information that can be visualized by a display (e.g., a display system configured to be positioned toward the eyes of a user wearing the wearable device (101), including the first display (110-1) and the second display (110-2) of FIG. 1), as information that can be processed by the perception HAL (281), the system HAL (282), and / or the XR HAL (283). By using the information, the wearable device (101) can perform foveated rendering by controlling the display system of the wearable device (101).

[0091] Hereinafter, the operation of a wearable device (101) and / or a processor (210) performing foveated rendering is described with reference to FIG. 3a and / or FIG. 3b.

[0092] FIGS. 3A and 3B illustrate flowcharts of a wearable device according to various embodiments of the disclosure. The wearable device (101) of FIGS. 1 and / or 2A and / or the processor (210) of FIG. 2A may perform at least one of the operations of FIG. 3A. For example, at least some of the operations of FIG. 3A may be performed by a wearable device that executes the foveated renderer (290) of FIG. 2A. For example, at least some of the operations of FIG. 3A may be performed by the processor (210) and / or the DPU (213) of FIG. 2A. The order in which the operations of FIG. 3A are performed is not limited to the order illustrated in FIG. 3A. For example, the processor of the wearable device may perform the operations of FIG. 3A in a different order than the order illustrated in FIG. For example, a processor of a wearable device may perform at least two of the operations of FIG. 3A substantially simultaneously.

[0093] Referring to FIG. 3A, in operation (310), a processor of a wearable device according to one embodiment may activate foveated rendering based on the execution of a software application. For example, the processor may activate foveated rendering of operation (310) based on the execution of a software application configured to display static content (e.g., a gallery application for viewing photos). For example, the processor may detect parameters and / or flags related to foveated rendering from a resource of a file for the software application (e.g., a package file). The resource may include a manifest (e.g., an extended marked-up language (XML) file having a specified file name and / or a specified extension, such as "manifest.xml"). Upon detecting the specified parameters and / or specified flags indicating that foveated rendering is permitted from the resource, the processor may activate foveated rendering of operation (310). Based on the activation of the foveated rendering of operation (310), the processor can perform operations subsequent to operation (310) of FIG. 3a.

[0094] Referring to FIG. 3A, in operation (320), a processor of a wearable device according to one embodiment may obtain information about a gaze position. The processor may obtain information about a gaze position of operation (320) using sensor data of a sensor (220) of FIG. 2A (e.g., an image sensor (130) and / or a motion sensor (222)). For example, the information about the gaze position may include coordinate values ​​of the gaze position based on a coordinate system associated with a display (e.g., a display (110) of FIG. 1 and / or FIG. 2A). For example, the information about the gaze position may include a speed, velocity, and / or displacement at the gaze position indicated by the coordinate values. For example, the information about the gaze position may include a direction, velocity, and / or speed in which the gaze position moves, referred to as a motion vector. For example, the magnitude of the motion vector may correspond to a speed (e.g., a moving speed) of the gaze position. The operation of a wearable device that obtains information on the gaze position of an action (320) is described with reference to FIG. 4.

[0095] Referring to FIG. 3A, in operation (330), the processor of the wearable device according to one embodiment may determine whether the movement speed of the gaze position indicated by the information of operation (320) is included in a first speed range. The first speed range may be formed to determine the movement speed of the gaze position greater than a first threshold speed. The processor that detects a speed included in the first speed range (330-Yes) may perform operation (332). The processor that detects a speed different from the first speed range (e.g., a speed less than the first speed range and / or the first threshold speed) (330-No) may perform operation (340). For example, if the movement speed of the gaze position indicated by the information of operation (320) is too fast, the processor may perform operation (332).

[0096] Referring to FIG. 3A, in operation (332), a processor of a wearable device according to one embodiment may perform foveated rendering using an image having a first resolution among an image having a first resolution or an image having a second resolution exceeding the first resolution. The processor may display or fill the image having the first resolution in the entire foveated portion and a peripheral portion of the display area. While detecting a movement speed of a gaze position included in a first speed range, the processor may perform operation (332) to stop displaying the image having the second resolution in the foveated portion. For example, while the user moves the gaze position to a portion other than the foveated portion, the movement speed of the gaze position may increase to the first speed range. In the above example, the processor may stop displaying the image having the second resolution (e.g., a high-resolution image) in the foveated portion and only perform rendering of the image having the first resolution (e.g., a low-resolution image). Because rendering of high-resolution images is omitted, the processor performing the action (332) can react to shifts in gaze position at a faster rate.

[0097] Referring to FIG. 3A, in operation (340), the processor of the wearable device according to an embodiment of the disclosure may determine whether the movement speed (e.g., speed) of the gaze position indicated by the information of operation (320) falls within a second speed range less than a first speed range. The second speed range may be configured to determine a movement speed of the gaze position that is lower than a first threshold speed. The second speed range may be configured to determine a movement speed of the gaze position that is lower than the first threshold speed and greater than a second threshold speed that is lower than the first threshold speed. The processor that detects the movement speed of the gaze position that falls within the second speed range (340—Yes) may perform at least one of operations (342, 344). The processor that detects a movement speed that is different from the second speed range (e.g., a movement speed less than the second threshold speed) (340—No) may perform operation (350).

[0098] Referring to FIG. 3A, in operation (342), according to one embodiment, a processor of a wearable device may determine a frame rate using a movement speed and / or direction of a gaze position. If the direction of the gaze position is maintained within a specified angular range for a specified period of time, the processor may determine a frame rate based on the movement speed of the gaze position. For example, if the gaze position moves at a specified movement speed along a specified direction, the processor may determine a frame rate based on the movement speed of the gaze position. For example, the processor may determine a frame rate for the foveated portion such that an image fixed on the foveated portion is displayed while the gaze position moves within the foveated portion. For example, as the movement speed of the gaze position decreases, the frame rate determined by operation (342) may decrease. A frame rate determined by a processor that detects a gaze position moving at a constant speed along a specified direction is exemplarily described with reference to FIG. 5. For example, if the gaze position moves in an irregular direction within the second speed range, the processor may determine the frame rate of the operation (342) at a specified frame rate (e.g., the frame rate at which the foveated rendering of the operation (332) is performed).

[0099] In one embodiment, the processor performing operation (342) can change the size of the foveated portion along with the frame rate of operation (342). For example, the size of the foveated portion can be changed inversely proportional to the movement speed of the gaze position. For example, as the movement speed of the gaze position increases, the processor can decrease the size of the foveated portion. The size of the foveated portion can be increased in proportion to the movement speed of the gaze position to maintain the frame rate or to reduce the amount of increase in the frame rate. The gaze position can be changed not only by the movement of the pupils, but also by motions of the user that are different from the pupils (e.g., rotation of at least a part of the user's body, such as the head).

[0100] Referring to FIG. 3A, in operation (344), a processor of a wearable device according to an embodiment of the disclosure may perform foveated rendering using an image of a second resolution obtained according to a frame rate determined by operation (342). The processor performing operation (344) may perform rendering of a foveated portion using an image of a second resolution obtained according to a frame rate determined by operation (342). The processor may perform rendering of a peripheral portion using an image having a first resolution lower than the second resolution. Rendering of the peripheral portion may be performed according to a frame rate determined by operation (342), similar to rendering of the foveated portion.

[0101] Referring to FIG. 3A, in operation (350), a processor of a wearable device according to an embodiment of the disclosure may maintain foveated rendering using an image of a second resolution in response to a movement speed of a gaze position included in a third speed range less than a second speed range. The third speed range may be formed to determine a movement speed of the gaze position that is slower than the second speed range. The third speed range may be set to determine a movement speed of the gaze position that is less than a second threshold speed. The processor that detects a movement speed of the gaze position included in the third speed range may perform operation (350). In operation (350), the processor may maintain displaying an image of the second resolution on the foveated portion.

[0102] For example, when the movement speed of the gaze position is lowered from the second speed range to the third speed range, the processor may perform rendering of the peripheral portion at a frame rate determined by operation (342), while performing rendering of the foveated portion at a frame rate lower than the frame rate determined by operation (342).

[0103] For example, the processor may maintain displaying a single image displayed in the foveated portion. The processor may display the single image in the foveated portion using operation (350) until the gaze position moving within the third speed range reaches the boundary line of the foveated portion. The processor, upon detecting the gaze position reaching the boundary line of the foveated portion, may change the position and / or size of the foveated portion. The processor may re-acquire or generate an image of the second resolution based on the changed position and / or changed size of the foveated portion.

[0104] As described above, the processor may, in response to a translation speed of the gaze position that is higher than the first threshold speed, stop controlling the display using an image having a second resolution lower than the first resolution to fill the entire display area of ​​the display with an image having the first resolution. The processor may, in response to a translation speed of the gaze position that is lower than the first threshold speed and higher than the second threshold speed, control the display to display an image in the foveated portion and display an image in the peripheral portion using the direction in which the gaze position moves and / or the translation speed and frame rate of the gaze position. The processor may, in response to a translation speed of the gaze position that is lower than the second threshold speed, control the display to display a single image in the foveated portion.

[0105] FIG. 3B illustrates the operation of a wearable device according to gaze movement. At operation (360), a processor of the wearable device according to an embodiment of the disclosure may activate foveated rendering. Operation (360) of FIG. 3B may be performed similarly to operation (310) of FIG. 3A. At operation (362), a processor of the wearable device according to an embodiment of the disclosure may obtain information about gaze position. Operation (362) of FIG. 3B may be performed similarly to operation (320) of FIG. 3A. For example, the processor may determine or calculate a speed and / or direction of movement of the gaze position using a sensor for detecting motion of the wearable device, such as an IMU, and / or a gaze vector. Based on the movement speed and / or direction of the gaze position indicated by the information of the action (362), the processor of the wearable device can selectively perform at least one of the actions (364, 368, 372, 380) of FIG. 3b.

[0106] In operation (364) of FIG. 3B, the processor of the wearable device may detect a rapid gaze movement. For example, the processor may detect a movement speed of the gaze position that is faster than a specified reference speed. Upon detecting a movement speed of the gaze position that is faster than the reference speed, the processor of the wearable device may perform operation (366). In operation (366), according to an embodiment of the disclosure, the processor of the wearable device may perform foveated rendering based on a first mode. The foveated rendering based on the first mode may correspond to the foveated rendering of operation (332) of FIG. 3A. An operation of the wearable device performing foveated rendering based on the first mode of operation (366) is described with reference to FIG. 7A and / or FIG. 7B.

[0107] In operation (368) of FIG. 3B, the processor of the wearable device may detect a static gaze movement. For example, the processor may detect a movement speed of the gaze position that is slower than a specified reference speed. For example, the processor may detect a movement speed of the gaze position that is substantially zero. In operation (368), the processor may detect a movement speed of the gaze position that remains substantially zero for a specified period of time. Having detected the static gaze movement, the processor of the wearable device may perform operation (370). In operation (370), according to an embodiment of the disclosure, the processor of the wearable device may perform foveated rendering based on a second mode. The foveated rendering based on the second mode may correspond to the foveated rendering of operation (350) of FIG. 3A. The operation of a wearable device performing foveated rendering based on the second mode of operation (370) is described with reference to FIG. 8a and / or FIG. 8b.

[0108] In operation (372) of FIG. 3B, the processor of the wearable device may detect regular gaze movement. For example, the processor may detect a movement of a gaze position according to a constant movement direction and / or a constant movement speed. The processor of the wearable device that detects the regular gaze movement of operation (372) may perform operation (374). In operation (374), the processor of the wearable device according to an embodiment of the disclosure may perform foveated rendering based on a third mode. The foveated rendering based on the third mode may correspond to the foveated rendering of operations (342, 344) of FIG. 3A. The operation of the wearable device that performs foveated rendering based on the third mode of FIG. 3B is described with reference to FIG. 9A and / or FIG. 9B.

[0109] In operation (380) of FIG. 3B, the processor of the wearable device may detect a gaze movement that does not match any of the conditions of operations (364, 368, 372). In operation (380), the processor of the wearable device may perform operation (382). In operation (382), the processor of the wearable device may perform rendering based on a default mode, according to an embodiment of the disclosure. An operation of the wearable device performing foveated rendering based on the default mode of FIG. 3B is described with reference to FIG. 6A and / or FIG. 6B.

[0110] Below, with reference to FIG. 4, the operation of the wearable device performed to obtain information on the gaze position of the operation (320) is described.

[0111] FIG. 4 illustrates the operation of a wearable device for determining information about a gaze position, according to one embodiment of the disclosure. The operation of the wearable device of FIG. 4 may be performed by the wearable device (101) of FIG. 1 and FIG. 2A and / or the processor (210) of FIG. 2A. The operation of the wearable device of FIG. 4 may be related to at least one of the operations of FIG. 3A (e.g., operation (320)).

[0112] Referring to FIG. 4, a state of a wearable device (101) worn on the head of a user (410) is illustrated. Within the state of FIG. 4, the wearable device (101) can obtain information on a gaze position using a sensor (e.g., sensor (220) of FIG. 2A). For example, the wearable device (101) can detect the motion of the head of the user (410) wearing the wearable device (101) using a motion sensor (e.g., motion sensor (222) of FIG. 2A). Using sensor data of the motion sensor, the wearable device (101) can obtain information indicating the motion of the head of the user (410). The information may include a vector (dh) indicating the direction of the head of the wearable device (101) and / or the user (410). The above information may be obtained or generated based on the execution of the location tracker (271) of FIG. 2a.

[0113] According to one embodiment of the disclosure, the wearable device (101) can acquire images and / or videos of two eyes (e.g., a left eye (140-1) and / or a right eye (140-2)) of a user (410) wearing the wearable device (101) using an image sensor (e.g., an image sensor (130) of FIG. 2A). For example, from a first gaze tracking camera (130-1) configured to be positioned toward the left eye (140-1), the wearable device (101) can acquire an image and / or video of the left eye (140-1). From the image and / or video of the left eye (140-1), the wearable device (101) can calculate or determine a vector (dl) representing the direction of the left eye (140-1). For example, from a second gaze tracking camera (130-2) configured to be positioned toward the right eye (140-2), the wearable device (101) can acquire an image and / or video of the right eye (140-2). From the image and / or video of the right eye (140-2), the wearable device (101) can calculate or determine a vector (dr) indicating the direction of the right eye (140-2). The vectors (dl, dr) can be acquired or generated based on the execution of the gaze tracker (274) of FIG. 2A.

[0114] According to one embodiment of the disclosure, a wearable device (101) can calculate or determine information about a gaze position (g) using at least one of the vectors (dl, dr, dh). The information about the gaze position (g) can be determined based on a combination of the vectors (dl, dr, dh). For example, when the wearable device (101) processes the position of a virtual object (e.g., virtual objects (421, 422) and / or a real object) using a coordinate system linked to an external space, the wearable device (101) can determine a gaze position (g) within the coordinate system using vectors (dl, dr, dh). The wearable device (101) can determine a virtual object (421) included in a screen (120) to be displayed on a display (e.g., a display (110) of FIG. 2A) using the coordinate system. For example, by comparing the positions (v1, v2) of the virtual objects (421, 422) within the coordinate system and a portion within the coordinate system corresponding to the screen (120), the wearable device (101) can display the virtual object (421) within the screen (120).

[0115] Referring to FIG. 4, in a state where the first display (110-1) is arranged to cover the left eye (140-1) and the second display (110-2) is arranged to cover the right eye (140-2), the display area formed by the first display (110-1) and / or the second display (110-2) can cover or obscure the field of view of the user (410). The wearable device (101) can use information about the gaze position (g) to determine the foveated portion and the surrounding portion overlapping with the gaze position (g) within the display area where the screen (120) is displayed. Depending on the movement speed of the gaze position (g), the wearable device (101) can change the frame rate for rendering a high-quality image for the foveated portion, thereby reducing the resources of the wearable device (101) occupied for rendering the high-quality image.

[0116] Below, with reference to FIG. 5, the operation of a wearable device (101) that changes the frame rate according to the movement speed of the gaze position (g) is described.

[0117] FIG. 5 illustrates the operation of a wearable device for determining a frame rate based on the movement speed of a gaze position according to one embodiment of the disclosure. The operation of the wearable device of FIG. 5 may be performed by the wearable device (101) of FIG. 1 and FIG. 2A and / or the processor (210) of FIG. 2A. The operation of the wearable device of FIG. 5 may be related to at least one of the operations of FIG. 3A.

[0118] Referring to FIG. 5, a graph is shown that illustrates the relationship between the movement speed of a gaze position and the frame rate. The x-axis of the graph of FIG. 5 may be related to the movement speed of the gaze position, and the y-axis may be related to the frame rate for foveated rendering. Within the graph of FIG. 5, speed ranges (e.g., a first speed range (501), a second speed range (502), a third speed range (503), and a fourth speed range (504)) are shown, which are distinguished by a first threshold speed (vth1), a second threshold speed (vth2), and a third threshold speed (vth3). The first speed range (501) of FIG. 5 may correspond to the first speed range of the operation (330) of FIG. 3a, the second speed range (502) may correspond to the second speed range of the operation (340) of FIG. 3a, and the third speed range (503) may correspond to the third speed range of the operation (350).

[0119] A wearable device that detects a moving speed within a first speed range (501) can perform foveated rendering based on the first mode of operation (366) of FIG. 3B. A wearable device that detects a moving speed within a third speed range (503) can perform foveated rendering based on the second mode of operation (370) of FIG. 3B. A wearable device that detects a moving speed within a second speed range (502) can perform foveated rendering based on the third mode of operation (374) of FIG. 3B. A wearable device that detects a moving speed within a fourth speed range (504) can perform foveated rendering based on the default mode of operation (382) of FIG. 3B.

[0120] Referring to FIG. 5, line (511) may represent a frame rate associated with foveated rendering of an image of a first resolution. The image of the first resolution may be acquired for display in a peripheral portion and / or the entire display area of ​​a display (e.g., display (110) of FIG. 2A). Line (512) may represent a frame rate associated with foveated rendering of an image of a second resolution lower than the first resolution. The image of the second resolution may be acquired for display in the foveated portion.

[0121] Referring to FIG. 5, while identifying a movement speed of a gaze position included in a first speed range (501) higher than a first threshold speed (vth1) (or a first reference speed), the processor may display an image on the entire display area of ​​the display according to a first frame rate (f1) indicated by a line (511). Within the first speed range (501), the processor may perform foveated rendering based on the operation (332) of FIG. 3A. For example, the processor may display an image having a first resolution, acquired according to the first frame rate (f1), on the entire display area. While detecting a movement speed of a gaze position higher than the first threshold speed (vth1), the processor may stop displaying another image different from the image having the first resolution (e.g., an image having a second resolution lower than the first resolution) in the foveated portion. For example, the processor may not acquire another image different from the image having the first resolution. For example, the processor may discard other images that are different from the image having the first resolution. The operations of the processor performed for foveated rendering while detecting the movement speed of the gaze position exceeding the first threshold speed (vth1) are described with reference to FIG. 7A and / or FIG. 7B.

[0122] Referring to FIG. 5, while identifying a movement speed of a gaze position that is lower than a first threshold speed (vth1) (or a third threshold speed (vth3) lower than the first threshold speed (vth1)) and higher than a second threshold speed (vth2) (or a second reference speed) (e.g., while identifying a movement speed of a gaze position included in a second speed range (502)), the processor can display an image in a peripheral portion, as indicated by a line (511), at a frame rate lower than the first frame rate (f1), and can display an image in a foveated portion, as indicated by a line (512), at a frame rate lower than the first frame rate (f1). For example, within the second speed range (502), the frame rates indicated by the lines (511, 512) can be substantially equally varied depending on the movement speed of the gaze position. For example, within the second speed range (502), when the movement speed of the gaze position decreases, the frame rates used to display images in the foveated portion and the peripheral portion, respectively, may decrease. For example, within the second speed range (502), when the movement speed of the gaze position increases, the frame rates used to display images in the foveated portion and the peripheral portion, respectively, may increase.

[0123] For example, if the speed of movement of the gaze position is reduced, the period of time during which the gaze position moves within a foveated portion of a certain size may be increased. During this period, the wearable device may display a single high-resolution image within the foveated portion, and may allocate resources allocated to acquire or generate the high-resolution image to perform other functions. When performing foveated rendering using a single high-resolution image, the wearable device may generate or determine an image to be displayed on the display from the high-resolution image using extrapolation.

[0124] The frame rate within the second speed range (502) may be related to the direction and / or movement speed of the gaze position detected by the wearable device. For example, if the gaze position moves along a specific direction at a constant movement speed (e.g., within a specified error range and / or a speed change less than a specified deviation), the wearable device may change the frame rate according to lines (511, 512). For example, if the gaze position moves at an irregular movement speed and / or in an irregular direction, the wearable device may display an image in the foveated portion and display an image in the peripheral portion according to a frame rate (e.g., the first frame rate (f1)) that is independent of the lines (511, 512). The operation of the wearable device to display an image in the foveated portion and display an image in the peripheral portion according to the first frame rate (f1) is described with reference to FIGS. 6A and 6B.

[0125] Referring to FIG. 5, while identifying a movement speed of a gaze position that is lower than a first threshold speed (vth1) and higher than a third threshold speed (vth3) (e.g., while identifying a movement speed of a gaze position included in a fourth speed range (504)), the processor can display images in the peripheral portion and the foveated portion according to a first frame rate (f1) indicated by lines (511, 512). For example, within the fourth speed range (504), the frame rates indicated by lines (511, 512) can be maintained at a designated frame rate, such as the first frame rate (f1). The first frame rate (f1) can be a frame rate of a display system of the wearable device (e.g., a combination of the first display (110-1) and / or the second display (110-2) of FIG. 1).

[0126] Referring to FIG. 5, while identifying a movement speed of a gaze position within a third speed range (503) lower than a second threshold speed (vth2), the processor may display an image in the foveated portion and an image in the peripheral portion according to a frame rate (f2) indicated by line (512). The embodiment is not limited thereto, and the wearable device may display an image in the peripheral portion according to a frame rate higher than the frame rate (f2). For example, the wearable device may maintain displaying a single high-resolution image in the foveated portion to maintain or assist the user's focus on the foveated portion based on detecting movement of a static gaze position. For example, the processor may stop or bypass acquiring further high-resolution images while displaying the single high-resolution image. For example, the processor may discard other high-resolution images generated based on the execution of a software application while displaying the single high-resolution image.

[0127] According to one embodiment of the disclosure, the wearable device can determine the size of the foveated portion in response to the movement speed of the gaze position included in the second speed range (502) and / or the third speed range (503), as a size related to the movement speed. In one embodiment of the disclosure, as the movement speed increases, the size of the foveated portion can decrease. As the movement speed decreases, the size of the foveated portion can increase. When the movement speed increases above the first threshold speed (vth1), the foveated portion can be substantially eliminated. Using the determined size, the wearable device can acquire or generate a high-resolution image from a software application.

[0128] As described above with reference to FIG. 5, a wearable device according to an embodiment of the disclosure can determine a level and / or mode of foveated rendering based on a movement speed of a gaze position. The wearable device can determine a parameter for stabilizing an image displayed on a screen (e.g., an image stabilizing parameter) in conjunction with a level of foveated rendering (e.g., the speed ranges of FIG. 5). The parameter for stabilizing the image can include a dynamic parameter and / or a static parameter. The dynamic parameter can be changed or determined based on a movement speed and / or a movement direction of the gaze position.

[0129] Hereinafter, operations of a wearable device for determining a frame rate in each of the speed ranges (e.g., a first speed range (501), a second speed range (502), and a third speed range (503)) are described with reference to FIGS. 6a, 6b, 7a, 7b, 8a, 8b, 9a, and 9b.

[0130] FIGS. 6A and 6B illustrate flowcharts of operations of a wearable device related to foveated rendering according to various embodiments of the disclosure. The wearable device (101) of FIGS. 1 and / or 2A and / or the processor (210) of FIG. 2A may perform at least one of the operations of FIG. 6A. For example, at least some of the operations of FIG. 6A may be performed by a wearable device executing the foveated renderer (290) of FIG. 2A. For example, at least some of the operations of FIG. 6A may be performed by the processor (210) and / or the DPU (213) of FIG. 2A. The operations of FIG. 6A may be related to at least one of the operations of FIG. 3A. The order in which the operations of FIG. 6A are performed is not limited to the order illustrated in FIG. For example, the processor of the wearable device may perform the operations of FIG. 6A in a different order than the order illustrated in FIG. 6A. For example, the processor of the wearable device may perform at least two of the operations of FIG. 6A substantially simultaneously.

[0131] Referring to FIG. 6A, within operation (610), a processor of a wearable device according to an embodiment of the disclosure may acquire a first image (611) and a second image (612) corresponding to a portion (619) of the first image (611). The portion (619) may correspond to a foveated portion within a display area (e.g., portion (129) of FIG. 1) that overlaps with a gaze position. A width (w1) and a height (h1) of the first image (611) may be smaller than a width (wd) and a height (hd) of the entire display area of ​​a display (e.g., display (110) of FIG. 2A), respectively. A width (w2) and a height (h2) of the second image (612) may be smaller than a width and a height of the entire display area of ​​the display, respectively. The width (w2) and height (h2) of the second image (612) may match the width (w1) and height (h1) of the first image (611), respectively. The first resolution of the first image (611) may be lower than the second resolution of the second image (612).

[0132] Referring to FIG. 6A, within operation (620), a processor of a wearable device according to an embodiment of the disclosure may enlarge a first image (611) using the size of a display area of ​​a display. Referring to FIG. 6A, an enlarged first image (621) is illustrated. The processor may enlarge the first image (611) using upscaling to have a width (wd) and height (hd) of the entire display area of ​​the display. The enlarged first image (621) may have a lower resolution than the first resolution of the first image (611).

[0133] Referring to FIG. 6A, within operation (630), a processor of a wearable device according to an embodiment of the disclosure may obtain a composite image (631) by combining an enlarged first image (621) and a second image (612). The second image (612) may be composited onto a portion of the composite image (621) that is mapped to a portion (619) of the first image (611) that corresponds to the second image (612). Referring to FIG. 6A, a composite image (631) obtained by performing operation (630) is exemplarily illustrated. A portion (639) of the composite image (631) may correspond to a portion (619) of the first image (611). For example, a location of the portion (639) combined with the second image (612) within the composite image (631) may correspond to a location of the portion (619) within the first image (611). In other words, the portion (639) of the composite image (631) may correspond to the foveated portion. The position of the portion (639) within the composite image (631) may correspond to a gaze position within the display area, or may include a gaze position within the display area. For example, the processor may use a visual effect, such as blur, to reduce the visibility of the boundary between the second image (612) and the enlarged first image (621).

[0134] Referring to FIG. 6A, within operation (640), a processor of a wearable device according to an embodiment of the disclosure may control at least one display to display a composite image (631). Since the width (wd) and height (hd) of the composite image (631) correspond to the width and height of the entire display area, the processor may display the composite image (631) over the entire display area. A portion (639) of the composite image (631), into which a second image (612) having a lower resolution than the first resolution of the first image (611) is combined, may have a higher resolution than the other portions. Accordingly, when the display is controlled to display the composite image (631), the resolution of the foveated portion (e.g., portion (639)) may be higher than the resolution of the surrounding portions.

[0135] Referring to FIG. 6B, foveated rendering performed by a processor such as the DPU (213) of FIG. 2A is exemplarily illustrated. The DPU can obtain a first image (611) and a second image (612) corresponding to a portion (619) of the first image (611) from a host (600) including an application processor (e.g., CPU (211) of FIG. 2A). The host (600) may include a processor such as the application processor, as well as a software application executed by the processor. The DPU can generate or obtain an enlarged first image (621) by enlarging the first image (611) based on upscaling. The DPU can perform a synchronization and merging operation (650) to combine the enlarged first image (621) and the second image (612). Based on the above combination, the DPU can generate a composite image (631). The composite image (631) can have a form in which at least a portion of the second image (612) is superimposed on the enlarged first image (621). For example, the second image (612) can be positioned on a portion (639) of the composite image (631). The DPU can control a display system to display the composite image (631).

[0136] FIGS. 7A and 7B illustrate flowcharts of operations of a wearable device that detects a gaze position moving at a speed within a first speed range (501) of FIG. 5, according to various embodiments of the disclosure. The wearable device (101) of FIGS. 1 and / or 2A and / or the processor (210) of FIG. 2A may perform at least one of the operations of FIG. 7A. For example, at least some of the operations of FIG. 7A may be performed by a wearable device executing the foveated renderer (290) of FIG. 2A. For example, at least some of the operations of FIG. 7A may be performed by the processor (210) and / or the DPU (213) of FIG. 2A. The operation of FIG. 7A may be related to at least one of the operations of FIG. 3A (e.g., operation (332)). The order in which the operations of FIG. 7A are performed is not limited to the order illustrated in FIG. 7A. For example, the processor of the wearable device may perform the operations of FIG. 7A in a different order than the order illustrated in FIG. 7A. For example, the processor of the wearable device may perform at least two of the operations of FIG. 7A substantially simultaneously.

[0137] When a user wearing a wearable device rapidly moves his or her head or gaze, distortion (e.g., blur) may occur in the foveated portion due to foveated rendering of the foveated portion. To reduce the distortion, according to one embodiment of the disclosure, the wearable device may perform foveated rendering based on a low-resolution image (e.g., the first image (711) of FIG. 7A and / or FIG. 7B ) without a high-resolution image.

[0138] Referring to FIG. 7A, in operation (710), according to one embodiment of the disclosure, a processor of a wearable device may acquire a first image (711) having a first resolution among a first resolution or a second resolution greater than the first resolution. The first image (711) may be acquired from a software application executed by the wearable device to display the entire display area and / or a peripheral portion of a display (e.g., display (110) of FIG. 2A). For example, a width (w1) and a height (h1) of the first image (711) may be smaller than a width (wd) and a height (hd) of the entire display area of ​​the display, respectively.

[0139] Referring to FIG. 7A, in operation (720), according to one embodiment of the disclosure, a processor of a wearable device may enlarge (e.g., upscale) a first image (711) using the size of a display area of ​​a display. For example, the processor may enlarge (e.g., upscale) the first image (711) based on a setting value of a software application. For example, the processor may increase the width (w1) of the first image (711) to the width (wd) of the entire display area, and the height (h1) of the first image (711) to the height (hd) of the entire display area using upscaling. Referring to FIG. 7A, an enlarged first image (721) based on operation (720) is illustrated. The enlarged first image (721) may have a resolution that is lower than or equal to the first resolution of the first image (711).

[0140] Referring to FIG. 7A, within operation (730), a processor of a wearable device according to an embodiment of the disclosure may control at least one display to display an enlarged first image (721). The processor may control at least one display to fill the entire display area with the enlarged first image (721).

[0141] As described above with reference to FIG. 3A and / or FIG. 5, when the processor detects a movement speed of the gaze position that is higher than the first threshold speed and / or the first reference speed, the processor may perform the operations of FIG. 7A. The processor performing the operations of FIG. 7A may control the display using only the first image (711) without acquiring any image corresponding to a portion (719) of the first image (711) corresponding to the foveated portion (e.g., another image having a higher resolution than the first resolution of the first image (711). When the gaze position moves rapidly, motion blur may occur in the high-resolution image due to the rapid movement of the gaze position. In order not to visualize the motion blur included in the high-resolution image, the processor may control the display using only the low-resolution image among the low-resolution image and the high-resolution image, as described above with reference to FIG. 7A.

[0142] Referring to FIG. 7B, foveated rendering performed by a processor such as the DPU (213) of FIG. 2A is illustrated. The DPU can obtain a first image (711) and a second image (712) corresponding to a portion (719) of the first image (711) from a host (600) including an application processor (e.g., CPU (211) of FIG. 2A). The DPU can generate or obtain an enlarged first image (721) by enlarging the first image (711) based on upscaling. Referring to FIG. 7B, the DPU can perform a synchronization and merge operation (750) without the second image (712). The DPU can control a display system to display the enlarged first image (721). For example, for a synchronization and merge operation (750), only the enlarged first image (721) may be used among the enlarged first image (721) and the second image (712). The DPU may upscale the enlarged first image (721) and display the upscaled, enlarged first image (721).

[0143] FIGS. 8A and 8B illustrate flowcharts of operations of a wearable device that detects a gaze position moving at a speed within a third speed range (503) of FIG. 5, according to various embodiments of the disclosure. The wearable device (101) of FIGS. 1 and / or 2A and / or the processor (210) of FIG. 2A may perform at least one of the operations of FIG. 8A. For example, at least some of the operations of FIG. 8A may be performed by a wearable device executing the foveated renderer (290) of FIG. 2A. For example, at least some of the operations of FIG. 8A may be performed by the processor (210) and / or the DPU (213) of FIG. 2A. The operation of FIG. 8A may be related to at least one of the operations of FIG. 3A (e.g., operation (350)). The order in which the operations of FIG. 8A are performed is not limited to the order illustrated in FIG. 8A. For example, the processor of the wearable device may perform the operations of FIG. 8A in a different order than the order illustrated in FIG. 8A. For example, the processor of the wearable device may perform at least two of the operations of FIG. 8A substantially simultaneously.

[0144] Referring to FIG. 8A, in operation (810), a processor of a wearable device according to an embodiment of the disclosure may acquire a plurality of images, and a second image (812) corresponding to a portion (819) of a first image (811) among the plurality of first images. The plurality of images may include downscaled images (e.g., the first image (811) and / or the third image (813)). The portion (819) may correspond to a foveated portion within a display area. The second image (812) corresponding to the portion (819) may be a high-resolution single foveated image used for foveated rendering. A width (w1) and a height (h1) of the first image (811) may be smaller than a width (wd) and a height (hd) of the entire display area of ​​a display (e.g., the display (110) of FIG. 2A), respectively. The width (w1) and the height (h1) of the first image (811) may be smaller than the width and height of the source image corresponding to the first image (811), respectively. The resolution of the first image (811) may be lower than the resolution of the source image. The width (w2) and the height (h2) of the second image (812) may be smaller than the width and the height of the entire display area of ​​the display, respectively. The width (w2) and the height (h2) of the second image (812) may match the width (w1) and the height (h1) of the first image (811), respectively. The first resolution of the first image (811) may be lower than the second resolution of the second image (812).

[0145] Referring to FIG. 8A, within operation (820), a processor of a wearable device according to an embodiment of the disclosure may control at least one display to display a first composite image (821) obtained by performing foveated rendering based on a first image (811) and a second image (812). Operation (820) of FIG. 8A for generating the first composite image (821) may be performed similarly to operations (620, 630, 640) of FIG. 6A. For example, the processor may enlarge the first image (811) according to the size of the entire display area. By combining the second image (812) onto the enlarged first image (811), the processor may obtain or generate the first composite image (821). Within the first composite image (821), the second image (812) may have a position based on a portion (829) corresponding to the foveated portion and / or a position of the portion (819) within the first image (811). While displaying the first composite image (821), a user wearing the wearable device (101) may clearly view the second image (812) through the portion (829) corresponding to the foveated portion. When enlarging a plurality of images including the first image (811), the processor may not enlargage the second image (812). For example, the size and / or resolution of the second image (812) may be maintained while performing foveated rendering.

[0146] Referring to FIG. 8A, within operation (830), a processor of a wearable device according to an embodiment of the disclosure may determine whether to retain the second image (812) for foveated rendering based on a movement speed of the gaze position. For example, if the movement speed of the gaze position falls within a relatively slow speed range (e.g., the third speed range (503) of FIG. 5), the processor may determine to retain the second image (812) for foveated rendering. If it is determined to retain the second image (812) for foveated rendering, the processor may perform operation (840). If it is determined to retain the second image (812) for foveated rendering, the processor may not acquire another image after the second image (812) having a second resolution higher than the first resolution.

[0147] Referring to FIG. 8A, within operation (840), a processor of a wearable device according to an embodiment of the disclosure may control at least one display to display a second composite image (841) obtained by performing foveated rendering based on a third image (831) and a second image (812) based on the determination of operation (830). For example, the processor may enlarge the third image (831) according to the size of the entire display area. On the enlarged third image (831) having a width (wd) and a height (hd) of the entire display area, the processor may combine the second image (812) to generate or obtain a second composite image (841). Within the second composite image (841), the second image (812) may be positioned on a portion (849) corresponding to a foveated portion. While displaying the second composite image (841), a user wearing the wearable device (101) can continue to view the second image (812) through the portion (849) corresponding to the foveated portion.

[0148] As described above with reference to FIGS. 3A and / or 5, if the processor detects a movement speed of the gaze position that is lower than the second threshold speed, the processor may perform the operations of FIG. 8A. The processor performing the operations of FIG. 8A may display a single image (e.g., the second image (812)) in the foveated portion and change the image (e.g., the first image (811)) displayed in the peripheral portion. For example, the frame rate corresponding to the foveated portion may be reduced to substantially zero. Since the processor performs foveated rendering using a single high-resolution image, resources (e.g., memory bandwidth) occupied for obtaining the high-resolution image may be saved. The second image (812) may continue to be displayed in frames displayed on the display according to the refresh rate.

[0149] Referring to FIG. 8B, foveated rendering performed by a processor such as the DPU (213) of FIG. 2A is exemplarily illustrated. The DPU can obtain a plurality of images (e.g., the first image (811), the third image (813)), including a first image (811) downscaled from a source image, from a host (600) including an application processor (e.g., the CPU (211) of FIG. 2A). The DPU can obtain a second image (812) corresponding to a portion (819) of the first image (811) from the host (600).

[0150] The DPU may generate or obtain an enlarged first image (811-1) and an enlarged third image (813-1) by enlarging each of the first image (811) and the third image (813) based on upscaling. Referring to FIG. 8B, the DPU may perform a synchronization and merging operation (850) to combine the second image (812) with each of the enlarged images (e.g., the enlarged first image (811-1) and the enlarged third image (813-1)). Using the synchronization and merging operation (850), at least a portion of the second image (812) may be combined onto the enlarged first image (811-1). For example, a portion of the second image (812) may be positioned on a portion (869) of a composite image (860) generated from the enlarged first image (811-1). The DPU can control the display system to display the composite image (860). The second image (812) can be combined on the enlarged first image (811-1) and / or the enlarged third image (813-1) without any enlargement operation such as upscaling.

[0151] FIGS. 9A and 9B illustrate flowcharts of operations of a wearable device that detects a gaze position moving at a speed within the second speed range (502) of FIG. 5, according to various embodiments. The wearable device (101) of FIGS. 1 and / or 2A and / or the processor (210) of FIG. 2A may perform at least one of the operations of FIG. 9A. For example, at least some of the operations of FIG. 9A may be performed by a wearable device that executes the foveated renderer (290) of FIG. 2A. For example, at least some of the operations of FIG. 9A may be performed by the processor (210) and / or the DPU (213) of FIG. 2A. The operation of FIG. 9A may be related to at least one of the operations of FIG. 3A (e.g., operations (342, 344)). The order in which the operations of FIG. 9A are performed is not limited to the order illustrated in FIG. 9A. For example, the processor of the wearable device may perform the operations of FIG. 9A in a different order than the order illustrated in FIG. 9A. For example, the processor of the wearable device may perform at least two of the operations of FIG. 9A substantially simultaneously.

[0152] Referring to FIG. 9A, within operation (910), a processor of a wearable device according to an embodiment of the disclosure may acquire a plurality of images. The plurality of images may include scaled-down images (e.g., a first image (911) and a third image (941)) and images corresponding to a foveated portion (e.g., a second image (912) and a fourth image (942)).

[0153] Referring to FIG. 9A, within operation (920), a processor of a wearable device according to an embodiment of the disclosure may control at least one display (e.g., display (110) of FIG. 2A) to display a first composite image (921) obtained by performing foveated rendering based on a first image (911) and a second image (912). The second image (912) may correspond to a portion (919) of the first image (911). Operation (920) performed by the processor to obtain the first composite image (921) may be performed similarly to operation (820) of FIG. 8A. For example, the first composite image (921) may include a portion (929) corresponding to the foveated portion. The first composite image (921) may include the first image (911) enlarged according to the size of the entire display area. The embodiment is not limited thereto, and the first composite image (921) may have a size defined by a size larger than the size of the first image (911) (e.g., a size predetermined by the size of the display and / or a software application executed by the processor). A second image (912) may be combined with a portion (929) of the first composite image (921).

[0154] In one embodiment of the disclosure, while controlling at least one display using the first composite image (921), the processor may perform frame extrapolation based on late stage restoration (LSR) to generate or display an intermediate image between the first composite image (921) and a composite image that was displayed before the first composite image (921).

[0155] Referring to FIG. 9A, within operation (930), according to one embodiment of the disclosure, a processor of a wearable device may determine a frame rate using a speed and / or direction (dg) of movement of a gaze position (g) overlapping a portion of a display area associated with a second image (912). The processor may determine the frame rate such that, in response to the speed of movement of the gaze position (g) included in a second speed range (e.g., the second speed range (502) of FIG. 5), an update of the second image (912) is performed when the gaze position (g) reaches a boundary of a portion of the display area where the second image (912) is displayed.

[0156] For example, the processor may change or determine a frame rate to maintain displaying the second image (912) based on the first composite image (921) while the gaze position (g) is within a portion of the display area where the second image (912) is displayed. For example, the processor may maintain controlling the display using the first composite image (921) that includes the second image (912) while the gaze position (g) moves along the direction (dg) within a portion (929) of the first composite image (921) that is combined with the second image (912). For example, the processor may maintain controlling the display using the first composite image (921) that includes the second image (912) until the gaze position (g) reaches a boundary line between the portion (929) and the other portion (928) along the direction (dg).

[0157] Referring to FIG. 9A, within operation (940), a processor of a wearable device according to an embodiment of the disclosure may acquire a fourth image (942) corresponding to a portion (949) of a third image (941) subsequent to the first image (911) using the frame rate of operation (930). The third image (941) may have a width (w1) that is smaller than the width of the entire display area and a height (h1) that is smaller than the height of the entire display area. Each of the third image (941) and the fourth image (942) may have the same dimensions (e.g., width and / or height) as the first image (911) and the second image (912). The processor may acquire the third image (941) and the fourth image (942) according to the frame rate determined by operation (930). For example, at a frame rate proportional to the movement speed of the gaze position, the processor can acquire a third image (931) and a fourth image (942).

[0158] Referring to FIG. 9A, a third image (941) and a fourth image (942) acquired by a processor performing operation (940) are exemplarily illustrated. The processor can determine a portion (949) of the third image (941) corresponding to another portion (928) reached by a gaze position (g) within the first composite image (921). For example, the portion (949) may correspond to or include a gaze position overlapping a display area. The portion (949) may be adjacent to a portion (919) of the first image (911) along a direction (dg) of the gaze position (g), or may have a position next to the portion (919). Within the first composite image (921), a portion (928) corresponding to a portion (949) of the third image (941) may be positioned next to a portion (929) corresponding to a portion (919) of the first image (911), depending on the direction (dg) in which the gaze position (g) moves.

[0159] Referring to FIG. 9A, within operation (950), a processor of a wearable device according to an embodiment of the disclosure may control at least one display to display a second composite image (951) obtained by performing foveated rendering based on a third image (941) and a fourth image (942). The processor may perform operation (950) similarly to operations (620, 630, 640) of FIG. 6A using the third image (941) and the fourth image (942). For example, the processor may enlarge the third image (941) according to the size of the entire display area. The processor may generate or obtain the second composite image (951) by combining the fourth image (942) on an enlarged portion (949) within the enlarged third image (941). The processor can control at least one display to fill the entire display area with the second composite image (951).

[0160] Referring to FIG. 9B, foveated rendering performed by a processor, such as the DPU (213) of FIG. 2A, is exemplarily illustrated. The DPU can obtain, from a host (600) including an application processor (e.g., CPU (211) of FIG. 2A), a first set (913) of multiple images, including a first image (911) downscaled from a source image. The DPU can obtain, from the host (600), a second image (912) corresponding to a portion (919) of the first image (911). The DPU can obtain, from the host (600), a second set (914) of multiple images, corresponding to a foveated portion, such as the second image (912).

[0161] The DPU may acquire a third set (915) of enlarged images by enlarging a plurality of images (e.g., the first image (911)) included in the first set (913) based on upscaling. Referring to FIG. 9B, the third set (915) may include the enlarged first image (911-1). The width (wd) and height (hd) of the first image (911-1) may be determined by properties set for upscaling. The properties may be related to the size of the display and / or display area of ​​the wearable device. The properties may be related to a software application executed by the wearable device.

[0162] The DPU can perform a synchronization and merge operation (950) to combine the images included in the second set (914) with the enlarged images included in the third set (915). Based on the synchronization and merge operation (950), the DPU can obtain a fourth set (916) of composite images. Using the synchronization and merge operation (950), the second image (912) can be combined on the enlarged first image (911-1). For example, the second image (912) can be positioned on a portion (929) of the composite image (921) included in the fourth set (916). As described above with reference to FIG. 9A, since the frame rate is changed, the DPU can obtain or generate the fourth set (916) of composite images according to the changed frame rate.

[0163] The DPU may perform extrapolation based on the LSR (960) on the fourth set (916) of synthetic images to obtain a fifth set (961) of synthetic images based on the frame rate of the display. The fifth set (961) may further include intermediate images based on the frame rate of the display in addition to the synthetic images (921) included in the fourth set (916). The DPU may control the display of the wearable device to display the synthetic images of the fifth set (961).

[0164] As described above, according to one embodiment of the disclosure, a wearable device can at least temporarily stop performing foveated rendering using a high-resolution image, or change a frame rate at which foveated rendering is performed using a high-resolution image. The wearable device can change the frame rate at which foveated rendering is performed using a high-resolution image depending on a movement speed and / or direction of a gaze position. For example, in order to save resources occupied for performing foveated rendering using a high-resolution image, the processor can change the frame rate depending on a movement speed of the gaze position.

[0165] Hereinafter, with reference to FIGS. 10A and / or 10B, an appearance of a wearable device (101) described with reference to FIGS. 1, 2A, 2B, 3A, 3B, 4, 5, 6A, 6B, 7A, 7B, 8A, 8B, and 9A is illustrated. The wearable device (1000) of FIGS. 10A and / or 10B may be an example of the wearable device (101) of FIG. 1.

[0166] FIGS. 10A and 10B illustrate examples of the appearance of a wearable device according to various embodiments of the disclosure. The wearable device (1000) of FIGS. 10A and 10B may include at least a portion of the hardware of the wearable device (1000) described with reference to FIGS. 1 and / or 2A. An example of the appearance of a first side (1010) of a housing of the wearable device (1000) according to an embodiment of the disclosure is illustrated in FIG. 10A, and an example of the appearance of a second side (1020) opposite to the first side (1010) may be illustrated in FIG. 10B.

[0167] Referring to FIG. 10A, a first surface (1010) of a wearable device (1000) according to an embodiment of the disclosure may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1000) may further include a strap and / or one or more temples for being fixed to a body part of the user. A first display (1050-1) for outputting an image to a left eye among the user's two eyes, and a second display (1050-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (1010). The wearable device (1000) is formed on the first surface (1010) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (1050-1) and the second display (1050-2). The first display (1050-1) and the second display (1050-2) may correspond to the first display (110-1) and the second display (110-2) of FIG. 1, respectively.

[0168] According to one embodiment of the disclosure, a wearable device (1000) may include cameras (1060-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (1050-1) and the second display (1050-2). The cameras (1060-1) may correspond to the gaze tracking camera (e.g., the first gaze tracking camera (130-1) and / or the second gaze tracking camera (130-2)) of FIG. 1. According to one embodiment of the disclosure, a wearable device (1000) may include cameras (1060-5, 1060-6) for photographing and / or recognizing a face of a user. The cameras (1060-5, 1060-6) may be referred to as FT cameras. The wearable device (1000) can control an avatar representing the user in a virtual space based on the facial motion of the user identified using cameras (1060-5, 1060-6). For example, the wearable device (1000) can change the texture and / or shape of a portion of the avatar (e.g., a portion of the avatar representing a human face) using information obtained by cameras (1060-5, 1060-6) (e.g., FT cameras) and representing the facial expression of the user wearing the wearable device (1000).

[0169] Referring to FIG. 10B, a camera (e.g., cameras (1060-7, 1060-8, 1060-9, 1060-10, 1060-11, 1060-12)) and / or a sensor (e.g., a depth sensor (1030)) for obtaining information related to the external environment of the wearable device (1000) may be disposed on a second surface (1020) opposite to the first surface (1010) of FIG. 10A. For example, the cameras (1060-7, 1060-8, 1060-9, 1060-10) may be disposed on the second surface (1020) to recognize external objects.

[0170] For example, using cameras (1060-11, 1060-12), the wearable device (1000) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1060-11) can be placed on the second face (1020) of the wearable device (1000) to obtain an image to be displayed through the second display (1050-2) corresponding to the right eye among the two eyes. The camera (1060-12) can be placed on the second face (1020) of the wearable device (1000) to obtain an image to be displayed through the first display (1050-1) corresponding to the left eye among the two eyes.

[0171] According to one embodiment of the disclosure, a wearable device (1000) may include a depth sensor (1030) disposed on a second face (1020) to identify a distance between the wearable device (1000) and an external object. Using the depth sensor (1030), the wearable device (1000) may obtain spatial information (e.g., a depth map) for at least a portion of a field of view (FoV) of a user wearing the wearable device (1000). Although not shown, a microphone may be disposed on the second face (1020) of the wearable device (1000) to obtain a sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0172] In one embodiment of the disclosure, a method for reducing and / or optimizing resources occupied for foveated rendering may be required. In one embodiment of the disclosure, a method for changing a frame rate for acquiring images for foveated rendering depending on a movement speed of a gaze position may be required. As described above, according to one embodiment of the disclosure, a wearable device (e.g., wearable device (101) of FIG. 1 , wearable device (1000) of FIG. 10A and / or FIG. 10B ) may include a display system including a first display and a second display, each of which is configured to be positioned toward each of eyes of a user wearing the wearable device, at least one sensor (e.g., sensor (220) of FIG. 2A ), at least one processor including a processing circuit (e.g., processor (210) of FIG. 2A ), and a memory including one or more storage media for storing instructions (e.g., memory (215) of FIG. 2A ). The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to obtain information about a gaze position using the at least one sensor. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to obtain a plurality of first images corresponding to a display area of ​​the display system based on identifying, using the information, a movement speed of the gaze position that is slower than a reference speed. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to obtain a second image corresponding to a foveated portion specified within the display area based on the gaze position.The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to perform foveated rendering on a screen to be displayed through the display area by combining the second image with each of the plurality of first images that have been upscaled based on a size of the display area.

[0173] For example, the reference speed may be a first reference speed. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to identify whether the movement speed and movement direction of the gaze position are maintained based on identifying the movement speed of the gaze position that is faster than the first reference speed and slower than a second reference speed that is higher than the first reference speed. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to determine a frame rate that is within a range less than a reference frame rate of the display system and that corresponds to the movement speed, based on identifying that the movement speed and movement direction are maintained. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to acquire the plurality of first images corresponding to the display area of ​​the display system according to the frame rate. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to acquire the plurality of third images corresponding to the foveated portion according to the frame rate. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to combine the plurality of third images, each of which is upscaled based on a size of the display area, with each of the plurality of first images.The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to perform foveated rendering on a screen to be displayed through the display area by performing extrapolation of combinations of the plurality of first images and the plurality of third images based on a difference between the reference frame rate and the frame rate.

[0174] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to acquire the plurality of third images arranged along the movement direction within the display area.

[0175] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to determine the frame rate such that the foveated rendering is performed based on the fourth image corresponding to the foveated portion while the gaze position overlaps the foveated portion corresponding to the fourth image among the plurality of third images.

[0176] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to determine sizes of the plurality of third images based on identifying a movement speed of the gaze position that is faster than the first reference speed and slower than the second reference speed.

[0177] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to acquire the plurality of first images and the plurality of third images according to the reference frame rate from among the frame rate and the reference frame rate, based on identifying a change in at least one of the movement speed or the movement direction.

[0178] For example, the reference speed may be a first reference speed. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to acquire the plurality of first images corresponding to the display area of ​​the display system based on identifying the movement speed of the gaze position that is faster than a second reference speed that is higher than the first reference speed. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to perform the foveated rendering on the screen using the plurality of first images upscaled based on the size of the display area, such that portions of the plurality of first images are located in the foveated portion.

[0179] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to acquire the plurality of first images having sizes less than a size of the entire display area.

[0180] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to obtain the information using sensor data of the at least one sensor, the at least one sensor including an image sensor configured to be positioned toward the user's eye (e.g., image sensor (130) of FIG. 1), and a motion sensor configured to detect motion of the wearable device (e.g., motion sensor (222) of FIG. 2A).

[0181] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to update the second image to be used for the foveated rendering based on identifying that the gaze location overlaps a portion of the display area corresponding to a boundary of the second image.

[0182] According to one embodiment of the disclosure, a method of a wearable device as described above may be provided. The wearable device may include a display system including a first display and a second display, each of which is configured to be positioned toward the eyes of a user wearing the wearable device, and at least one sensor. The method may include an operation of obtaining information about a gaze position using the at least one sensor. The method may include an operation of obtaining a plurality of first images corresponding to a display area of ​​the display system based on identifying a movement speed of the gaze position that is slower than a reference speed using the information. The method may include an operation of obtaining a second image corresponding to a foveated portion specified within the display area based on the gaze position. The method may include an operation of performing foveated rendering on a screen to be displayed through the display area by combining the second image with each of the plurality of first images, which are upscaled based on a size of the display area.

[0183] For example, the reference speed may be a first reference speed. The method may include an operation of identifying whether the movement speed and the movement direction of the gaze position are maintained based on identifying the movement speed of the gaze position that is faster than the first reference speed and slower than a second reference speed that is higher than the first reference speed. The method may include an operation of determining a frame rate that is included in a range less than a reference frame rate of the display system and that corresponds to the movement speed, based on identifying that the movement speed and the movement direction are maintained. The method may include an operation of acquiring the plurality of first images corresponding to the display area of ​​the display system according to the frame rate. The method may include an operation of acquiring a plurality of third images corresponding to the foveated portion according to the frame rate. The method may include an operation of combining the plurality of third images, each of the plurality of first images upscaled based on a size of the display area. The method may include an operation of performing foveated rendering on a screen to be displayed through the display area by performing extrapolation of combinations of the plurality of first images and the plurality of third images according to a difference between the reference frame rate and the frame rate.

[0184] For example, the operation of acquiring the plurality of third images may include an operation of acquiring the plurality of third images arranged along the movement direction within the display area.

[0185] For example, the operation of determining the frame rate may include an operation of determining the frame rate such that the foveated rendering based on the fourth image corresponding to the foveated portion is performed while the gaze position overlaps the foveated portion corresponding to the fourth image among the plurality of third images.

[0186] For example, the method may include an operation of determining sizes of the plurality of third images based on the movement speed, based on identifying the movement speed of the gaze position that is faster than the first reference speed and slower than the second reference speed.

[0187] For example, the operation of acquiring the plurality of third images may include an operation of acquiring the plurality of first images and the plurality of third images according to the reference frame rate among the frame rate and the reference frame rate, based on identifying a change in at least one of the movement speed or the movement direction.

[0188] For example, the reference speed may be a first reference speed. The method may include an operation of acquiring the plurality of first images corresponding to the display area of ​​the display system based on identifying the movement speed of the gaze position that is faster than a second reference speed that is higher than the first reference speed. The method may include an operation of performing the foveated rendering for the screen using the plurality of first images upscaled based on the size of the display area such that portions of the plurality of first images are located in the foveated portion.

[0189] For example, the operation of acquiring the plurality of first images may include an operation of acquiring the plurality of first images having sizes less than the size of the entire display area.

[0190] According to one embodiment of the disclosure as described above, a wearable device (e.g., wearable device (101) of FIG. 1, wearable device (1000) of FIG. 10A and / or FIG. 10B) may include at least one display (e.g., display (110) of FIG. 2A), at least one sensor (e.g., sensor (220) of FIG. 2A), at least one processor (e.g., processor (210) of FIG. 2A) including processing circuitry, and a memory (e.g., memory (215) of FIG. 2A) including one or more storage media for storing instructions. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display an image on the entire display area of ​​the at least one display at a first frame rate while identifying, through the at least one sensor, a movement speed of the gaze position that is higher than a first reference speed (e.g., the first reference speed (vth1) of FIG. 5 ). The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display an image on a foveated portion of the display area and to display an image on a peripheral portion of the display area at a second frame rate that is lower than the first frame rate while identifying, through the at least one sensor, a movement speed of the gaze position that is lower than the first reference speed and higher than a second reference speed (e.g., the second reference speed (vth2) of FIG. 5 ).The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display an image on the foveated portion at a third frame rate lower than the second frame rate, while identifying a movement speed of the gaze position lower than the second reference speed through the at least one sensor, and to display an image on the peripheral portion at the second frame rate. In one embodiment of the disclosure, the wearable device may reduce and / or optimize resources occupied for foveated rendering. In one embodiment of the disclosure, the wearable device may change a frame rate for acquiring images for foveated rendering depending on a movement speed of the gaze position.

[0191] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display an image in the foveated portion using the second frame rate, determined such that the image displayed in the foveated portion is updated when the gaze position reaches a boundary of the foveated portion while identifying a movement speed of the gaze position that is lower than the second reference speed through the at least one sensor.

[0192] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to acquire an image corresponding to another portion next to the portion corresponding to the foveated portion, within the image displayed in the peripheral portion, as an image to be displayed in the foveated portion when the gaze position reaches a boundary of the foveated portion.

[0193] For example, the other part may be positioned next to the part corresponding to the foveated part, depending on the direction in which the gaze position moves.

[0194] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to determine the third frame rate such that the image displayed in the foveated portion is maintained while the gaze position overlaps the foveated portion while identifying a movement speed of the gaze position that is lower than the second reference speed through the at least one sensor.

[0195] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to change the size of the foveated portion using a movement speed while identifying a movement speed of the gaze position that is lower than the first reference speed.

[0196] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display an image corresponding to a portion of the image at the first resolution in the foveated portion of the display area overlapping the gaze position, while displaying the image at the first resolution in the peripheral portion.

[0197] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display images having a size less than the size of the entire display area, in each of the foveated portion and the peripheral portion.

[0198] For example, the instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to obtain the information using sensor data of the at least one sensor, the at least one sensor including an image sensor configured to be positioned toward the user's eye (e.g., image sensor (130) of FIG. 2A), and a motion sensor configured to detect motion of the wearable device (e.g., motion sensor (222) of FIG. 2A).

[0199] In one embodiment of the disclosure as described above, a method of a wearable device including at least one display and at least one sensor may be provided. The method may include an operation of displaying an image on an entire display area of ​​the at least one display according to a first frame rate while identifying a movement speed of a gaze position higher than a first reference speed through the at least one sensor. The method may include an operation of displaying an image on a foveated portion of the display area according to a second frame rate lower than the first frame rate, and displaying the image on a peripheral portion of the display area while identifying a movement speed of a gaze position lower than the first reference speed and higher than a second reference speed through the at least one sensor. The method may include an operation of displaying an image on the foveated portion according to a third frame rate lower than the second frame rate, and displaying the image on the peripheral portion according to the second frame rate, while identifying a movement speed of a gaze position lower than the second reference speed through the at least one sensor.

[0200] For example, the displaying operation according to the second frame rate may include an operation of displaying an image in the foveated portion using the second frame rate, wherein the image displayed in the foveated portion is updated when the gaze position reaches a boundary of the foveated portion while identifying a movement speed of the gaze position that is lower than the second reference speed through the at least one sensor.

[0201] For example, the displaying operation according to the second frame rate may include an operation of acquiring an image corresponding to another part next to the part corresponding to the foveated part, within the image displayed in the peripheral part, as an image to be displayed in the foveated part when the gaze position reaches the boundary of the foveated part.

[0202] For example, the other part may be positioned next to the part corresponding to the foveated part, depending on the direction in which the gaze position moves.

[0203] For example, the displaying operation according to the third frame rate may include an operation of determining the third frame rate such that the image displayed in the foveated portion is maintained while the gaze position overlaps the foveated portion while identifying a movement speed of the gaze position lower than the second reference speed through the at least one sensor.

[0204] For example, the method may include an operation of changing the size of the foveated portion using the movement speed while identifying a movement speed of the gaze position that is lower than the first reference speed.

[0205] For example, the method may include an operation of displaying an image corresponding to a portion of the image of the first resolution in the foveated portion of the display area overlapping the gaze position while displaying an image of the first resolution in the peripheral portion.

[0206] For example, the method may include an operation of displaying images having a size smaller than the size of the entire display area in each of the foveated portion and the peripheral portion.

[0207] For example, the method may further include obtaining the movement speed using sensor data of at least one sensor, the sensor including an image sensor configured to be positioned toward the user's eye, and a motion sensor configured to detect motion of the wearable device.

[0208] In one embodiment of the disclosure as described above, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when executed by a wearable device comprising a display system including a first display and a second display, each of which is configured to be positioned toward each of eyes of a user wearing the wearable device, and at least one sensor, may cause the wearable device to control the display system such that images acquired according to a first frame rate are displayed on each of a foveated portion and a peripheral portion while identifying a movement speed of a gaze position higher than a reference speed through the at least one sensor. The instructions, when executed by the wearable device, may cause the wearable device to control the display system such that an image acquired according to the first frame rate is displayed in the peripheral portion and an image acquired according to a second frame rate lower than the first frame rate is displayed in the foveated portion while the wearable device identifies a movement speed of the gaze position lower than the reference speed through the at least one sensor.

[0209] For example, the instructions, when executed by the wearable device, may cause the wearable device to combine an image acquired according to the second frame rate with each of the images acquired according to the first frame rate to generate composite images to be displayed through the display system.

[0210] For example, the instructions, when executed by the wearable device, may cause the wearable device to control the display system such that an image acquired according to the first frame rate is displayed based on a size of a display area of ​​the display system while identifying a movement speed of the gaze position that is higher than another reference speed that exceeds the reference speed.

[0211] According to one embodiment of the disclosure as described above, a wearable device (e.g., wearable device (101) of FIG. 1, wearable device (1000) of FIG. 10A and / or FIG. 10B) may include a display system including a first display and a second display, each of which is configured to be positioned toward the eyes of a user wearing the wearable device, at least one sensor (e.g., sensor (220) of FIG. 2A), at least one processor including processing circuitry (e.g., processor (210) of FIG. 2A), and a memory including one or more storage media for storing instructions (e.g., memory (215) of FIG. 2A). The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to control the display system such that images acquired according to a first frame rate are displayed in the foveated portion and the peripheral portion, respectively, while the wearable device identifies, through the at least one sensor, a movement speed of the gaze position that is higher than the reference speed. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to control the display system such that images acquired according to the first frame rate are displayed in the peripheral portion and images acquired according to a second frame rate that is lower than the first frame rate are displayed in the foveated portion, while the wearable device identifies, through the at least one sensor, a movement speed of the gaze position that is lower than the reference speed.

[0212] For example, the instructions, when executed by a wearable device including at least one display and at least one sensor, may cause the wearable device to combine an image acquired according to the second frame rate with each of the images acquired according to the first frame rate to generate composite images to be displayed through the display system.

[0213] For example, the instructions, when executed by a wearable device including at least one display and at least one sensor, may cause the wearable device to control the display system such that an image acquired according to the first frame rate is displayed based on a size of a display area of ​​the display system while identifying a movement speed of the gaze position that is higher than another reference speed that exceeds the reference speed.

[0214] As used herein, the term "if" will be understood to mean "when, upon," "in response to determining," or "in response to detecting," depending on the context. Similarly, "if it is determined to," or "if [the stated condition or event] is detected," will optionally be understood to mean "upon determining," or "in response to determining," "upon detecting [the stated condition or event]," or "in response to detecting [the stated condition or event]."

[0215] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0216] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0217] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CDs (compact disc)-ROMs and DVDs (digital versatile discs), magneto-optical media such as floptical disks, and those configured to store program commands, including ROMs, RAMs, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0218] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0219] It is apparent that various embodiments of the disclosure according to the description and claims of the specification may be implemented in the form of hardware, software, or a combination of hardware and software.

[0220] Such software may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules), wherein the one or more programs include computer-executable instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method of disclosure.

[0221] Such software may be stored in the form of volatile or non-volatile storage, for example, a storage device such as ROM, whether erasable or overwritten, for example, RAM, memory chips, devices or integrated circuits, or optical or magnetic recording media such as CDs, DVDs, magnetic disks, or magnetic tapes, or the like. It is clear that the storage devices and storage media are various embodiments of non-transitory machine-readable storage suitable for storing a computer program or computer programs that, when executed, include instructions that implement various embodiments of the disclosure. Accordingly, various embodiments provide a program comprising code for implementing an apparatus or method as claimed in any of the claims of this disclosure, and a non-transitory machine-readable storage medium storing such a program.

[0222] While the disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure as defined by the appended claims and their equivalents.

Claims

1. In wearable devices, A display system including a first display and a second display, each of which is configured to be positioned toward each of the eyes of a user wearing the wearable device; At least one sensor; a memory storing one or more computer programs; and comprising one or more processors communicatively coupled with the display, the at least one sensor, and the memory; The one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: Obtaining information about the gaze position using at least one sensor, and Using the above information, based on identifying the movement speed of the gaze position that is slower than the reference speed: Acquire a plurality of first images corresponding to the display area of ​​the above display system, Acquire a second image corresponding to a foveated portion specified within the display area based on the above gaze position, and Computer-executable instructions for causing foveated rendering for a screen to be displayed through the display area to be performed by combining the second image with each of the plurality of first images upscaled based on the size of the display area, Wearable devices.

2. In claim 1, the reference speed is the first reference speed, and The one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: Based on identifying a speed of movement of said gaze position that is faster than said first reference speed and slower than a second reference speed that is higher than said first reference speed: Identify whether the movement speed and movement direction of the above gaze position are maintained, Based on identifying that the above movement speed and the above movement direction are maintained, a frame rate is determined that is within a range less than the reference frame rate of the display system and corresponds to the movement speed, According to the above frame rate, the plurality of first images corresponding to the display area of ​​the display system are acquired, According to the above frame rate, a plurality of third images corresponding to the foveated portion are acquired, Combining the plurality of third images with each of the plurality of first images upscaled based on the size of the display area, and Further comprising computer-executable instructions causing the computer to perform foveated rendering on a screen to be displayed through the display area by performing extrapolation of combinations of the plurality of first images and the plurality of third images according to a difference between the reference frame rate and the frame rate. Wearable devices.

3. In claim 2, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: Further comprising computer-executable instructions causing the computer to acquire a plurality of third images arranged along the movement direction within the display area. Wearable devices.

4. In claims 2 to 3, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: Further comprising computer-executable instructions causing the frame rate to be determined so that the foveated rendering based on the fourth image corresponding to the foveated portion is performed while the gaze position overlaps the foveated portion corresponding to the fourth image among the plurality of third images. Wearable devices.

5. In claims 2 to 4, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: Further comprising computer-executable instructions causing the sizes of the plurality of third images to be determined based on the movement speed of the gaze position, wherein the movement speed is faster than the first reference speed and slower than the second reference speed. Wearable devices.

6. In claims 2 to 5, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: Further comprising computer-executable instructions causing the computer to acquire the plurality of first images and the plurality of third images based on the frame rate and the reference frame rate, based on identifying at least one change in the movement speed or the movement direction. Wearable devices.

7. In claim 1, the reference speed is the first reference speed, and The one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: Based on identifying a speed of movement of said gaze position that is faster than a second reference speed that is higher than the first reference speed: Acquiring the plurality of first images corresponding to the display area of ​​the display system, and Further comprising computer-executable instructions causing said foveated rendering to be performed on said screen using said plurality of first images upscaled based on said size of said display area, such that portions of said plurality of first images are positioned on said foveated portion. Wearable devices.

8. In claims 1 to 7, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: Further comprising computer-executable instructions causing the computer to obtain a plurality of first images having sizes less than a size of the entire display area. Wearable devices.

9. In claims 1 to 8, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: An image sensor configured to be positioned toward the user's eye, and Further comprising computer-executable instructions causing the computer to obtain the information using sensor data of the at least one sensor including a motion sensor configured to detect motion of the wearable device. Wearable devices.

10. In claims 1 to 9, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the wearable device to: Further comprising computer-executable instructions causing the second image to be used for the foveated rendering to be updated based on identifying that the gaze position overlaps a portion of the display area corresponding to a boundary of the second image. Wearable devices.

11. A method of a wearable device comprising a display system including a first display and a second display, each of which is configured to be positioned toward each of the eyes of a user wearing the wearable device, and at least one sensor, An operation of obtaining information about a gaze position using at least one sensor; and Using the above information, based on identifying the movement speed of the gaze position that is slower than the reference speed: An operation of acquiring a plurality of first images corresponding to a display area of ​​the above display system; An operation of acquiring a second image corresponding to a foveated portion specified within the display area based on the above gaze position, and An operation of performing foveated rendering on a screen to be displayed through the display area by combining the second image with each of the plurality of first images upscaled based on the size of the display area, method.

12. In claim 11, the reference speed is the first reference speed, and The method is based on identifying a speed of movement of the gaze position that is faster than the first reference speed and slower than a second reference speed that is higher than the first reference speed: An action to identify whether the movement speed and movement direction of the above gaze position are maintained; An operation of determining a frame rate, which is included in a range less than a reference frame rate of the display system and corresponds to the movement speed, based on identifying that the movement speed and the movement direction are maintained; An operation of acquiring a plurality of first images corresponding to the display area of ​​the display system according to the frame rate; An operation of acquiring a plurality of third images corresponding to the foveated portion according to the above frame rate; An operation of combining each of the plurality of third images with each of the plurality of first images upscaled based on the size of the display area, and Further comprising an operation of performing foveated rendering on a screen to be displayed through the display area by performing extrapolation of combinations of the plurality of first images and the plurality of third images according to a difference between the reference frame rate and the frame rate. method.

13. In claim 12, the operation of acquiring the plurality of third images comprises: An operation of acquiring a plurality of third images arranged along the movement direction within the display area, method.

14. In claims 12 to 13, the operation of determining the frame rate comprises: Including an operation of determining the frame rate such that the foveated rendering based on the fourth image corresponding to the foveated portion is performed while the gaze position overlaps the foveated portion corresponding to the fourth image among the plurality of third images. method.

15. In claims 12 to 14, Further comprising an operation of determining sizes of the plurality of third images according to the movement speed based on identifying the movement speed of the gaze position, which is faster than the first reference speed and slower than the second reference speed. method.

Images