Electronic device, method, and non-transitory computer-readable storage medium for changing refresh rate

Abstract

This electronic device worn on the user's head may comprise: a memory storing instructions and including one or more storage media; one or more cameras; one or more sensors; a display assembly; and at least one processor including a processing circuit, wherein the instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: display, on the display assembly at a first refresh rate, at least some of images corresponding to a user's FOV acquired via the one or more cameras; maintain the refresh rate on the basis that a movement speed of the electronic device identified by using the one or more sensors is greater than a reference speed; change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate on the basis of identifying the movement speed lower than the reference speed for a reference time; and change the refresh rate back to the first refresh rate on the basis of detecting an external object moving toward the user.

Description

Electronic device, method, and non-transient computer-readable storage medium for changing the refresh rate

[0001] The present disclosure relates to an electronic device, a method, and a non-transient computer-readable storage medium for changing a refresh rate.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services by displaying computer-generated information in conjunction with external objects within the real world. The electronic device may be a head-mounted electronic device that can be worn by a user. For example, the electronic device may be AR glasses and / or a head-mounted device (HMD).

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] An electronic device worn on a user's head is described. The electronic device may include at least one processor comprising a processing circuit, one or more cameras, one or more sensors, a display assembly, and a memory comprising one or more storage media configured to store one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the electronic device to acquire images corresponding to the user's field of view (FOV) through the one or more cameras. The one or more programs may include instructions that cause the electronic device to display at least a portion of the images on the display assembly at a first refresh rate. The one or more programs may include instructions that cause the electronic device to identify the movement speed of the electronic device using the one or more sensors. The above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the first refresh rate, based on a movement speed greater than a reference speed while displaying at least a portion of the images at the first refresh rate. The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying a movement speed lower than the reference speed identified while displaying at least a portion of the images at the first refresh rate for a reference time.The above one or more programs may include instructions that cause the electronic device to change the refresh rate back to the first refresh rate based on detecting an external object moving toward the user using at least a portion of the images while displaying at least a portion of the images at the second refresh rate.

[0005] A method is described. The method may be performed within an electronic device worn on a user's head, comprising one or more cameras, one or more sensors, and a display assembly. The method may include the operation of acquiring images corresponding to the user's field of view (FOV) through the one or more cameras. The method may include the operation of displaying at least a portion of the images on the display assembly at a first refresh rate. The method may include the operation of identifying the movement speed of the electronic device using the one or more sensors. The method may include the operation of maintaining the refresh rate to display at least a portion of the images at the first refresh rate based on the movement speed being greater than a reference speed while displaying at least a portion of the images at the first refresh rate. The method may include the operation of changing the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate based on identifying the movement speed being lower than the reference speed identified while displaying at least a portion of the images at the first refresh rate for a reference time. The above method may include an operation of changing the refresh rate back to the first refresh rate based on detecting an external object moving toward the user using at least a portion of the images while displaying at least a portion of the images at the second refresh rate.

[0006] A non-transient computer-readable storage medium is described. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the electronic device to acquire images corresponding to the user's field of view (FOV) through the one or more cameras when executed by the electronic device worn on the user's head, the electronic device comprising one or more cameras, one or more sensors, and a display assembly. The one or more programs may include instructions that cause the electronic device to display at least a portion of the images on the display assembly at a first refresh rate when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify the movement speed of the electronic device using the one or more sensors when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the first refresh rate, based on a moving speed greater than a reference speed while displaying at least a portion of the images at the first refresh rate when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying, during a reference time, a moving speed lower than the reference speed identified while displaying at least a portion of the images at the first refresh rate when executed by the electronic device.The above one or more programs may include instructions that cause the electronic device to change the refresh rate back to the first refresh rate based on detecting an external object moving toward the user using at least a portion of the images while displaying at least a portion of the images at the second refresh rate when executed by the electronic device.

[0007] An electronic device worn on a user's head is described. The electronic device may include at least one processor comprising a processing circuit, one or more cameras, one or more sensors, a display assembly, and a memory comprising one or more storage media configured to store one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the electronic device to acquire images corresponding to the user's field of view (FOV) through the one or more cameras. The one or more programs may include instructions that cause the electronic device to display at least a portion of the images on the display assembly at a first refresh rate. The one or more programs may include instructions that cause the electronic device to identify the movement speed of the electronic device using the one or more sensors, and to identify whether an external object moving toward the user is detected using at least a portion of the images. The above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the first refresh rate, based on identifying the movement speed greater than the reference speed while displaying at least a portion of the images at the first refresh rate, or detecting the external object while displaying at least a portion of the images at the first refresh rate.The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying the movement speed lower than the reference speed during a reference time while displaying at least a portion of the images at the first refresh rate.

[0008] A method is described. The method may be performed within an electronic device worn on a user's head, comprising one or more cameras, one or more sensors, and a display assembly. The method may include the operation of acquiring images corresponding to the user's field of view (FOV) through the one or more cameras. The method may include the operation of displaying at least a portion of the images on the display assembly at a first refresh rate. The method may include the operation of identifying the movement speed of the electronic device using the one or more sensors, and identifying whether an external object moving toward the user is detected using at least a portion of the images. The method may include the operation of maintaining a refresh rate to display at least a portion of the images at the first refresh rate, based on identifying the movement speed greater than a reference speed while displaying at least a portion of the images at the first refresh rate, or detecting the external object while displaying at least a portion of the images at the first refresh rate. The above method may include an operation of changing the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying the movement speed lower than the reference speed during a reference time while displaying at least a portion of the images at the first refresh rate.

[0009] A non-transient computer-readable storage medium is described. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the electronic device to acquire images corresponding to the user's field of view (FOV) through the one or more cameras when executed by an electronic device worn on the user's head, the electronic device comprising one or more cameras, one or more sensors, and a display assembly. The one or more programs may include instructions that cause the electronic device to display at least a portion of the images on the display assembly at a first refresh rate. The one or more programs may include instructions that cause the electronic device to identify the movement speed of the electronic device using the one or more sensors and identify whether an external object moving toward the user is detected using at least a portion of the images when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the first refresh rate, based on identifying a moving speed greater than a reference speed while displaying at least a portion of the images at the first refresh rate, or detecting an external object while displaying at least a portion of the images at the first refresh rate, when executed by the electronic device.The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying the movement speed lower than the reference speed during a reference time while displaying at least a portion of the images at the first refresh rate when executed by the electronic device.

[0010] Figure 1 illustrates an example of a refresh rate required depending on the state of an electronic device.

[0011] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0012] FIG. 3 is a flowchart illustrating exemplary operations of an electronic device for determining whether to change the refresh rate according to the movement speed of the electronic device.

[0013] Figure 4 illustrates an example of determining whether to change the refresh rate according to the movement speed of the electronic device.

[0014] FIG. 5 is a flowchart illustrating exemplary operations of an electronic device for determining whether to change the refresh rate based on whether an external object moving toward a user is detected.

[0015] FIG. 6 illustrates an example of determining whether to change the playback rate based on whether an external object moving toward the user is detected.

[0016] FIG. 7 is a flowchart illustrating exemplary operations of an electronic device for determining the refresh rate as the user's gaze is directed toward the window.

[0017] FIG. 8 is a flowchart illustrating exemplary operations of an electronic device for determining a refresh rate based on whether the user's gaze is positioned on the window for a threshold time.

[0018] Figure 9 illustrates an example of determining the refresh rate based on whether the user's gaze is positioned on the window for a threshold time.

[0019] FIG. 10 illustrates an example in which an electronic device having the form of glasses displays at least a portion of images.

[0020] FIG. 11 is a block diagram of an electronic device in a network environment according to various embodiments.

[0021] FIG. 12a shows an example of a perspective view of a wearable device.

[0022] FIG. 12b shows an example of one or more hardware components placed within a wearable device.

[0023] FIGS. 13a and 13b show examples of the appearance of a wearable device.

[0024] Figure 14 shows an example of a block diagram of a wearable device.

[0025] Figure 15 shows an example of a block diagram of an electronic device for displaying an image in virtual space.

[0026] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0027] Figure 1 illustrates an example of a refresh rate required depending on the state of an electronic device.

[0028] Referring to FIG. 1, the electronic device (100) may include a head-mounted display (HMD) that can be worn on the head of a user (110). The electronic device (100) may be described as a head-mounted display (HMD) device, a headgear electronic device, a glasses-type (or goggle-type) electronic device, a video see-through (VST) or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device.

[0029] The electronic device (100) may include a display assembly (e.g., the display assembly (250) of FIG. 2). In a state (115), the electronic device (100) may display a window (125) in an environment (120) that is displayed (or provided) through the display assembly. For example, the environment (120) may include virtual reality (VR), augmented reality (AR), and / or an immersive environment. For example, the window (125) may be composed of text (or images). For example, the rate of change of state of the window (125) composed of text (or images) may be relatively low. The user (110) may not perceive a difference between the window (125) with a relatively low rate of change of state being displayed at a first refresh rate and being displayed at a second refresh rate lower than the first refresh rate. For example, a window (125) with a relatively low rate of change of state may be suitable for displaying at a relatively low second refresh rate. Displaying the window (125) at a first refresh rate may increase the power consumed within the electronic device (100). A solution may be required to resolve the problem of increased power consumption within the electronic device (100).

[0030] To resolve this problem, the electronic device (100) can display (or provide) the window (125) at a second refresh rate using less power than that consumed when displaying the window (125) at a relatively high first refresh rate. To determine the refresh rate for displaying the window (125), the gaze of the user (110) can be identified.

[0031] The electronic device (100) may further include one or more cameras (e.g., one or more cameras (230) of FIG. 2). In a state (130), the electronic device (100) may acquire images corresponding to the field of view (FOV) of the user (110) through the one or more cameras. The electronic device (100) may provide an environment (120) through the display assembly by displaying at least a portion of the images on the display assembly. The environment (120) may include an augmented reality (AR) and / or non-immersive environment. For example, in a state (130), an external object (135) may be moved toward the user (110). At least a portion of the images may include a visual object (140) corresponding to the external object (135) being moved toward the user (110). The electronic device (100) can identify an external object (135) moving toward the user (110) by using at least some of the images including a visual object (140).

[0032] For example, as an external object (135) moves toward the user (110), the user (110) and the external object (135) may collide. To prevent the risk of such collision to the user (110), it may be required that a visual object (140) corresponding to the external object (135) be displayed at a relatively high first refresh rate. As the visual object (140) is displayed at a second refresh rate lower than the first refresh rate, the external object (135) may be seen by the user (110) at a relatively low frequency. As the external object (135) is seen by the user (110) at a relatively low frequency, the user (110) may not perceive or prevent the collision. A method may be required to mitigate the risk to the user (110) caused by the visual object (140) being displayed at a relatively low second refresh rate.

[0033] To mitigate the risk to such a user (110), the electronic device (100) may use a variable refresh rate (VRR). The electronic device (100) may display a visual object (140) at a relatively high first refresh rate. The electronic device (100) may use the one or more cameras to identify an external object (135) moving toward the user (110) in order to determine the refresh rate for displaying the visual object (140).

[0034] The electronic device (100) may perform operations illustrated in the description of FIGS. 3 through 10 to determine a refresh rate for displaying at least a portion of the images. The electronic device (100) may include components for performing said operations. said components may be illustrated in the description of FIG. 2.

[0035] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0036] Referring to FIG. 2, the electronic device (200) may be described as a head-mount display (HMD) device wearable on a user's head, a headgear electronic device, a glasses-type (or goggle-type) electronic device, a video see-through (VST) or visible see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. The electronic device (200) may be worn on a user's head. An example of the structure of the electronic device (200) is described with reference to FIG. 12a, FIG. 12b, FIG. 13a and / or FIG. 13b. The electronic device (200) may include at least a part of the electronic device (1001) of FIG. 10 or correspond to at least a part of the electronic device (1001) of FIG. 10. The electronic device (200) may include at least one processor (210), memory (220), one or more cameras (230), one or more sensors (240), a display assembly (250), and / or one or more other cameras (260).

[0037] At least one processor (210) may include a processing circuit. At least one processor (210) may include a central processing unit (e.g., including a processing circuit). At least one processor (210) may include a graphic processing unit (e.g., including a processing circuit) and a neural processing unit (e.g., including a processing circuit). For example, at least one processor (210) may be configured to control a memory (220), one or more cameras (230), one or more sensors (240), a display assembly (250), and / or one or more other cameras (260). At least one processor (210) may be configured to execute instructions stored in the memory (220) individually or collectively to cause the electronic device (200) (or electronic device (100)) to perform at least some of the operations illustrated in the description of FIG. 1. At least one processor (210) may be configured to execute instructions stored in memory (220) individually or collectively to cause the electronic device (200) to perform at least some of the operations to be illustrated in the description of FIGS. 3 through 10.

[0038] The memory (220) may include one or more storage media. The memory (220) may store various data used by at least one component of the electronic device (200) (e.g., at least one processor (210), one or more cameras (230), one or more sensors (240), a display assembly (250), and / or one or more other cameras (260)). For example, the data may include input data or output data for software and related commands. The memory (220) may include volatile memory or non-volatile memory.

[0039] One or more cameras (230) may include one or more light sensors (e.g., a charged coupled device (CCD) sensor and / or a complementary metal oxide semiconductor (CMOS) sensor) that generate an electrical signal indicating the color and / or brightness of light. For example, one or more cameras (230) may be described as image sensors. For example, one or more cameras (230) may be available to acquire images of the space (or surrounding environment) in front of the electronic device (200). For example, one or more cameras (230) may have a first FOV. For example, at least a portion of one or more cameras (230) may have an FOV corresponding to the field of view (FOV) of the user's eye. For example, the FOV of a portion of one or more cameras (230) may be different from the FOV of another portion of one or more cameras (230).

[0040] One or more sensors (240) may be configured to identify the movement speed of the electronic device (200). One or more sensors (240) may identify changes in speed caused by the movement of the electronic device (200). For example, one or more sensors (240) may obtain depth information of the object by using the difference between the light emitted by light-emitting elements and the light received by light-receiving elements. For example, one or more sensors (240) may include an acceleration sensor or an accelerometer. One or more sensors (240) may include a gyro sensor. For example, one or more sensors (240) may be operably or operatively coupled with at least one processor (210).

[0041] The display assembly (250) may be configured to visualize information (or signals) provided from at least one processor (210). The display assembly (250) may be positioned toward the eyes of a user wearing the electronic device (200). The display assembly (250) may be configured to display at least a portion of images acquired through one or more cameras (230). The display assembly (250) may be configured to display windows and / or visual objects. The display assembly (250) may include at least one display.

[0042] One or more other cameras (260) may include one or more light sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate an electrical signal indicating the color and / or brightness of light. For example, one or more other cameras (260) may be described as image sensors. For example, one or more other cameras (260) may be arranged to face the eyes of the user of the electronic device (200). For example, each of one or more other cameras (260) may be available to acquire an image of the user's eyes of the electronic device (200). One or more other cameras (260) may be used to identify the user's gaze.

[0043] The electronic device (200) illustrated in the description of FIG. 2 may perform at least some of the operations illustrated in the descriptions of FIG. 3 through 10. The operations illustrated in the descriptions of FIG. 3 through 10 may be caused by (or within) the electronic device (200) under the control of at least one processor (210).

[0044] FIG. 3 is a flowchart illustrating exemplary operations of an electronic device for determining whether to change the refresh rate according to the movement speed of the electronic device.

[0045] Referring to FIG. 3, in operation 300, at least one processor (210) can acquire images corresponding to the user's field of view (FOV). For example, the user's FOV may correspond to the FOV of one or more cameras (230). For example, images corresponding to the user's FOV may be described as images of the FOV of one or more cameras (230). For example, images corresponding to the user's field of view (FOV) may be described as images of the space (or surrounding environment) in front of the electronic device (200).

[0046] In operation 310, at least one processor (210) may display at least a portion of the images on a display assembly (250). For example, the electronic device (200) may allow the user to see the real world (or physical environment, or actual environment) by displaying at least a portion of the images. For example, the electronic device (200) may display at least a portion of the images through a video see-through while providing a virtual space (or immersive environment) through the display assembly (250). For example, at least one processor (210) may display at least a portion of the images on the display assembly (250) to provide a video see-through as the user (or at least a portion of the user's body) moves out of a safety zone (or safety boundary).

[0047] For example, at least one processor (210) can display at least a portion of the images at a first refresh rate. For example, the first refresh rate can be described as the maximum refresh rate of the display assembly (250) (e.g., 120 Hz (Hertz)). For example, at least one processor (210) can display at least a portion of the images at a first refresh rate, which is the maximum refresh rate, for an unpredictable state.

[0048] In operation 320, at least one processor (210) can identify the movement speed of the electronic device (200) using one or more sensors (240). For example, the electronic device (200) may move as a user wearing the electronic device (200) moves. For example, at least one processor (210) can identify the movement speed of the electronic device (200) to prevent danger to the user caused by the user moving. For example, at least one processor (210) can distinguish between the user moving and the user not moving while only the head of the user wearing the electronic device (200) moves. For example, at least one processor (210) can distinguish between the user moving and the user making repetitive movements (e.g., the user sitting in a chair and making pendulum motions). For example, the user moving only the head (or the user making repetitive movements) may not cause danger to the user. For example, at least one processor (210) can identify the speed of movement of the electronic device (200) as the user moves to prevent danger to the user.

[0049] For example, at least one processor (210) can compare the movement speed of the measured electronic device (200) with a reference speed. For example, the reference speed may be described as a threshold speed set to determine whether the user's movement may cause danger to the user. For example, the reference speed may be predetermined. For example, the reference speed may be set by the user or changed.

[0050] In operation 330, at least one processor (210) may maintain a refresh rate to display at least a portion of the images at the first refresh rate based on the identified movement speed greater than the reference speed while displaying at least a portion of the images at the first refresh rate. For example, as the movement speed of the electronic device (200) is higher than the reference speed, at least one processor (210) may identify that the user is moving relatively fast. For example, as the user moves relatively fast, the user may collide with an external object. For example, at least one processor (210) may authorize the user to view changes in the external environment, including the external object, at a relatively high frequency to prevent a collision between the user and the external object. For example, at least one processor (210) may display at least a portion of the images at the first refresh rate, which is the maximum refresh rate (or less than the maximum refresh rate), in order to authorize the user to view changes in the external environment at a relatively high frequency.

[0051] According to another embodiment, at least one processor (210) can identify whether a user feels motion sick by using one or more other cameras (260). For example, at least one processor (210) can identify a user feeling motion sick based on prodromal signs of motion sickness (e.g., less blood pooling in the user's face, changes in the user's breathing, and / or the user sweating profusely). For example, at least one processor (210) can maintain a refresh rate to display at least a portion of the images at a first refresh rate based on identifying a user feeling motion sickness. For example, at least one processor (210) can alleviate the user feeling motion sickness by displaying at least a portion of the images at a first refresh rate, which is the maximum refresh rate.

[0052] In operation 340, at least one processor (210) may change the refresh rate from the first refresh rate to the second refresh rate based on a movement speed smaller than the reference speed identified while displaying at least a portion of the images at the first refresh rate (or based on identifying a movement speed smaller than the reference speed for a reference time). For example, the reference time may be predetermined or set (or changed) by the user. For example, the second refresh rate may be described as a refresh rate lower than the first refresh rate. For example, as the movement speed of the electronic device (200) is lower than the reference speed, at least one processor (210) may identify that the user is stopped or moving relatively slowly. For example, as the movement speed of the electronic device (200) is lower than the reference speed for more than a reference time, at least one processor (210) may identify that the user remains in a stopped state or the user remains moving slowly. For example, as the user remains stopped or moves relatively slowly, the user may not collide with an external object, or even if they do collide, the impact may not be significant. For example, at least one processor (210) may not cause danger to the user even if at least some of the images are displayed at a second refresh rate lower than the first refresh rate. For example, at least one processor (210) may reduce the power consumed within the electronic device (200) without causing danger to the user by changing the refresh rate from the first refresh rate to the second refresh rate. For example, at least one processor (210) may reduce the heat generated by the electronic device (200) or increase the battery life of the electronic device (200) by changing the refresh rate from the first refresh rate to the second refresh rate. Maintaining or changing the refresh rate according to the movement speed of the electronic device (200) is exemplified in the description of FIG. 4.

[0053] Figure 4 illustrates an example of determining whether to change the refresh rate according to the movement speed of the electronic device.

[0054] Referring to FIG. 4, the state (400) may be described as a state in which at least a portion of images acquired through one or more cameras (230) are displayed at a first refresh rate, which is the maximum refresh rate of the display assembly (250). For example, in the state (400), at least one processor (210) may provide augmented reality (410) (or mixed reality, or non-immersive environment) by displaying at least a portion of the images at the first refresh rate. For example, at least one processor (210) may identify the movement speed (415) of the electronic device (200) as the user (405) moves. For example, at least one processor (210) may compare the movement speed (415) of the electronic device (200) with a reference speed.

[0055] For example, at least one processor (210) may maintain a refresh rate to display at least some of the images at a first refresh rate, which is the maximum refresh rate, based on identifying a movement speed (415) of the electronic device (200) that is greater than the reference speed while displaying at least some of the images at a first refresh rate. For example, as the movement speed (415) of the electronic device (200) is higher than the reference speed, at least one processor (210) may identify that the user (405) is moving relatively fast. For example, as the user moves relatively fast, the user (405) may collide with an external object (420). For example, at least one processor (210) may authorize the user (405) to view the external object (420) at a relatively high frequency to prevent a collision between the user (405) and the external object (420). For example, at least one processor (210) may display at least a portion of images containing a visual object (425) corresponding to an external object (420) at a first refresh rate, which is the maximum refresh rate, in order to allow the user (405) to view the external object (420) at a relatively high frequency. For example, by displaying at least a portion of the images at a first refresh rate, which is the maximum refresh rate, the at least one processor (210) may cause the user (405) to avoid the external object (420).

[0056] For example, at least one processor (210) can change the refresh rate from the first refresh rate to the second refresh rate based on identifying a movement speed (415) of the electronic device (200) that is lower than the reference speed while displaying at least a portion of the images at the first refresh rate. For example, the second refresh rate may be described as a refresh rate lower than the first refresh rate. For example, as the movement speed (415) of the electronic device (200) is lower than the reference speed, at least one processor (210) can identify that the user (405) is stopped or moving relatively slowly. For example, as the user (405) is stopped or moving relatively slowly, the user (405) may not collide with an external object (420), or even if a collision occurs, the impact may not be significant. For example, at least one processor (210) may not cause danger to the user (405) even if it displays at least a portion of the images containing the visual object (425) at the second refresh rate lower than the first refresh rate. For example, at least one processor (210) can reduce power consumed within the electronic device (200) without causing danger to the user (405) by changing the refresh rate from a first refresh rate to a second refresh rate. For example, at least one processor (210) can reduce heat generation of the electronic device (200) or increase the battery life of the electronic device (200) by changing the refresh rate from a first refresh rate to a second refresh rate. For example, at least one processor (210) can detect an external object (420) moving toward the user (405) while displaying at least a portion of the images at the second refresh rate. Determining whether to change the refresh rate based on whether an external object (420) moving toward the user (405) is detected is exemplified in the description of FIG. 5.

[0057] FIG. 5 is a flowchart illustrating exemplary operations of an electronic device for determining whether to change the refresh rate based on whether an external object moving toward a user is detected.

[0058] Referring to FIG. 5, in operation 500, at least one processor (210) can change the refresh rate from a first refresh rate to a second refresh rate based on identifying the movement speed of an electronic device (200) that is smaller than a reference speed. For example, by changing the refresh rate from a first refresh rate to a second refresh rate, at least one processor (210) can display at least some of the images acquired through one or more cameras (230) at the second refresh rate. For example, operation 500 may correspond to operation 340 of FIG. 3.

[0059] In operation 510, at least one processor (210) can detect an external object moving toward the user by using at least a portion of the images. For example, the external object may be described as an external object located within the user's FOV. For example, at least one processor (210) can identify the direction of movement of the external object by comparing at least a portion of the images with each other. For example, at least one processor (210) can identify an external object approaching the electronic device (200) as an external object moving toward the user.

[0060] In operation 520, at least one processor (210) may change the refresh rate from the second refresh rate to the first refresh rate based on detecting an external object moving toward the user using at least one part of the images while displaying at least part of the images at the second refresh rate. For example, as the external object moves toward the user, the user may collide with the external object. For example, at least one processor (210) may allow the user to view changes in the external environment containing the external object at a relatively high frequency to prevent a collision between the user and the external object (or to cause the user to avoid the external object). For example, at least one processor (210) may display at least part of the images at the first refresh rate, which is the maximum refresh rate, to allow the user to view changes in the external environment at a relatively high frequency.

[0061] According to another embodiment, at least one processor (210) may change the refresh rate from the second refresh rate to the first refresh rate based on detecting an external object moving within a reference distance from the electronic device (200) using at least one portion of the images while displaying at least a portion of the images at the second refresh rate. For example, the reference distance may be described as a threshold distance from the electronic device (200) at which the external object and the user may collide as the external object is located within the reference distance. For example, the reference distance may be predetermined. For example, the reference distance may be set by the user or changed. For example, as the external object moves within the reference distance from the electronic device (200), the user may collide with the external object. For example, at least one processor (210) may authorize the user to view changes in the external environment containing the external object at a relatively high frequency to prevent a collision between the user and the external object (or to cause the user to avoid the external object). For example, at least one processor (210) can display at least some of the images at a first refresh rate, which is the maximum refresh rate, in order to allow the user to view changes in the external environment at a relatively high frequency.

[0062] In operation 530, at least one processor (210) may maintain a refresh rate to display at least a portion of the images at a second refresh rate based on identifying that no external object moving toward the user is detected using at least a portion of the images while displaying at least a portion of the images at a second refresh rate. For example, as no external object moving toward the user is detected, the user may not collide with the external object. For example, at least one processor (210) may not cause danger to the user even if it displays at least a portion of the images at a second refresh rate lower than the first refresh rate. For example, at least one processor (210) may reduce power consumed within the electronic device (200) while not causing danger to the user by maintaining a refresh rate to display at least a portion of the images at a second refresh rate. For example, at least one processor (210) may reduce heat generation of the electronic device (200) or increase the battery life of the electronic device (200) by maintaining a refresh rate to display at least a portion of the images at a second refresh rate. Maintaining or changing the playback rate depending on whether an external object moving toward the user is detected is exemplified in the description of FIG. 6.

[0063] FIG. 6 illustrates an example of determining whether to change the playback rate based on whether an external object moving toward the user is detected.

[0064] Referring to FIG. 6, the state (600) may be described as a state in which at least a portion of images acquired through one or more cameras (230) are displayed on a display assembly (250) at a second refresh rate. For example, in the state (600), at least one processor (210) may display at least a portion of the images at a second refresh rate based on the movement speed of the electronic device (200) being lower than a reference speed. For example, at least one processor (210) may provide augmented reality (410) (or mixed reality, or non-immersive environment) by displaying at least a portion of the images at a second refresh rate. For example, at least one processor (210) may use at least a portion of the images to detect an external object (605) moving toward a user (405).

[0065] For example, at least one processor (210) may change the refresh rate back from the second refresh rate to the first refresh rate, which is the maximum refresh rate, based on detecting an external object (605) moving toward the user (405) while at least some of the images are displayed at the second refresh rate. For example, as the external object (605) moves toward the user (405), the user (405) may collide with the external object (605) even if the user (405) does not move. For example, at least one processor (210) may allow the user (405) to see changes in the external environment containing the external object (605) at a relatively high frequency to prevent a collision between the user (405) and the external object (605) (or to cause the user (405) to avoid the external object (605). For example, at least one processor (210) can display a visual object (610) corresponding to an external object (605) that changes at a relatively high frequency to the user (405) within augmented reality (410). For example, at least one processor (210) can display at least a portion of images at a first refresh rate, which is the maximum refresh rate, in order to display a visual object (610) corresponding to an external object (605) that changes at a relatively high frequency.

[0066] For example, at least one processor (210) may maintain a refresh rate to display at least a portion of the images at a second refresh rate based on identifying that, while displaying at least a portion of the images at a second refresh rate, an external object (605) moving toward the user (405) is not detected using at least a portion of the images. For example, as the external object (605) moving toward the user (405) is not detected, the user (405) may not collide with the external object (605). For example, it may not be required to display a change in the visual object (610) corresponding to the external object (605) not moving toward the user (405) at a relatively high refresh rate. For example, even if at least one processor (210) displays at least a portion of the images at a second refresh rate lower than the first refresh rate, it may not cause danger to the user (405). For example, at least one processor (210) can reduce power consumed within the electronic device (200) without causing danger to the user (405) by maintaining a refresh rate to display at least a portion of the images at a second refresh rate. For example, at least one processor (210) can reduce heat generation of the electronic device (200) or increase the battery life of the electronic device (200) by maintaining a refresh rate to display at least a portion of the images at a second refresh rate. For example, at least one processor (210) can identify the user's (405) gaze using one or more other cameras (260) while displaying at least a portion of the images at a second refresh rate. Changing or maintaining the refresh rate according to the user's (405) gaze is exemplified in the description of FIG. 7.

[0067] FIG. 7 is a flowchart illustrating exemplary operations of an electronic device for determining the refresh rate as the user's gaze is directed toward the window.

[0068] Referring to FIG. 7, in operation 700, at least one processor (210) may maintain the refresh rate to display at least a portion of the images at the second refresh rate based on identifying that an external object moving toward the user is not detected using at least a portion of the images while displaying at least a portion of the images at the second refresh rate. For example, operation 700 may correspond to operation 530 of FIG. 5.

[0069] In operation 710, at least one processor (210) can identify the user's gaze using images of the user's eyes acquired through one or more other cameras (260). For example, at least one processor (210) can identify that the user's gaze is directed toward a window displayed in augmented reality (or mixed reality, or a non-immersive environment). For example, the window may be displayed on at least some of the images acquired through one or more cameras (230). For example, the window may be provided by a software application running in the electronic device (200).

[0070] In operation 720, at least one processor (210) may identify a third refresh rate that is predetermined for an application providing a window, based on identifying the user's gaze toward the window. For example, the third refresh rate may be predetermined based on the content provided by the application. For example, the rate of change of the state of a window composed of text (or images) may be relatively slow. As a non-limiting example, the third refresh rate may be predetermined as a refresh rate lower than the first refresh rate, which is the maximum refresh rate, for an application providing text content and / or image content. For example, the rate of change of the state of a window composed of video may be relatively fast. As a non-limiting example, the third refresh rate may be predetermined as a refresh rate corresponding to the first refresh rate, which is the maximum refresh rate, for an application providing video content. However, it is not limited thereto.

[0071] For example, at least one processor (210) can display at least a portion of images at a third refresh rate by maintaining the refresh rate at a second refresh rate based on a third refresh rate corresponding to a second refresh rate. For example, at least one processor (210) can display at least a portion of images at a third refresh rate by changing the refresh rate from a second refresh rate to a third refresh rate based on a third refresh rate different from a second refresh rate. For example, at least one processor (210) can reduce power consumption within the electronic device (200) by displaying a window with a relatively slow state change rate (e.g., a window composed of text and / or images) at a relatively low third refresh rate. For example, at least one processor (210) can reduce heat generation of the electronic device (200) or increase the battery life of the electronic device (200) by displaying a window with a relatively slow state change rate (e.g., a window composed of text and / or images) at a relatively low third refresh rate. For example, at least one processor (210) can provide a window that changes at a relatively high frequency to the user by displaying a window with a relatively fast rate of change of state (e.g., a window composed of video) at a relatively high third refresh rate. For example, even if the user's gaze is placed over the window, the user may not focus on the window. Maintaining or changing the refresh rate depending on whether the user focuses on the window is exemplified in the description of FIG. 8.

[0072] FIG. 8 is a flowchart illustrating exemplary operations of an electronic device for determining a refresh rate based on whether the user's gaze is positioned on the window for a threshold time.

[0073] Referring to FIG. 8, in operation 800, at least one processor (210) may display at least a portion of the images at a predetermined third refresh rate for the application providing the window, based on identifying the user's gaze toward the window. For example, operation 800 may correspond to operation 720 of FIG. 7.

[0074] For example, in operation 810, at least one processor (210) can identify the time during which the user's gaze is positioned on the window while displaying at least a portion of the images at a third refresh rate. For example, at least one processor (210) can identify whether the user's gaze is positioned on the window for a threshold time. For example, at least one processor (210) can determine whether the user is focusing on the window by identifying whether the user's gaze is positioned on the window for a threshold time. For example, the threshold time can be described as a reference time set to determine whether the user is focusing on the window. For example, the threshold time can be determined in advance. For example, the threshold time can be set by the user or changed.

[0075] In operation 820, at least one processor (210) may maintain a refresh rate to display at least a portion of images at a third refresh rate based on the user's gaze being positioned on the window for a threshold time. For example, as the user's gaze is positioned on the window for a threshold time, at least one processor (210) may determine that the user is focusing on the window. For example, as the user is focusing on the window, at least one processor (210) may maintain a refresh rate to display at least a portion of images at a third refresh rate suitable for the window.

[0076] According to another embodiment, at least one processor (210) can determine whether the user is focusing on the window based on the movement of the user's pupil. For example, at least one processor (210) can identify the movement of the user's pupil using one or more other cameras (260). For example, the movement of the user's pupil can be described as the movement of the user's central vision. For example, at least one processor (210) can determine that the user is focusing on the window based on identifying the movement of the user's pupil as a movement of viewing or reading content contained within the window. As an example without limitation, if the movement of the user's pupil is identified as a movement of reading text contained within the window, at least one processor (210) can determine that the user is focusing on the window. For example, at least one processor (210) can maintain a refresh rate to display at least a portion of the images at a third refresh rate suitable for the window based on determining that the user is focusing on the window using the movement of the user's pupil.

[0077] In operation 830, at least one processor (210) can identify a fourth refresh rate predetermined for another application providing another window located in an area adjacent to the window, based on a user's gaze that is not located on the window for a threshold time. For example, the other window being located in an area adjacent to the window can be described as at least a portion of the other window being located within a reference distance from the portion where the user's gaze is located on the window. For example, the fourth refresh rate may be predetermined based on the content provided by the other application.

[0078] According to another embodiment, at least one processor (210) can identify the trembling of the user's pupils and / or the user's facial expressions using one or more other cameras (260). For example, at least one processor (210) can determine whether the user is focusing on the window based on the trembling of the user's pupils and / or the user's facial expressions. For example, at least one processor (210) can determine that the user is not focusing on the window based on identifying the trembling of the user's pupils or identifying a preset facial expression of the user. For example, at least one processor (210) can determine that the user is not focusing on the window based on the trembling of the user's pupils and / or a preset facial expression of the user using a trained model.

[0079] For example, as the user's gaze is not positioned on the window for a threshold time, at least one processor (210) may determine that the user is not focusing on the window. For example, as the user is not focusing on the window, at least one processor (210) may be required to change a predetermined third refresh rate for the application providing the window. For example, as the user is not focusing on the window, another window located on an area adjacent to the window may be shown to the user. For example, at least one processor (210) may be required to display at least a portion of the images at a fourth refresh rate suitable for the other window. For example, at least one processor (210) may display at least a portion of the images at the fourth refresh rate by maintaining the refresh rate at the third refresh rate based on a fourth refresh rate corresponding to the third refresh rate. For example, at least one processor (210) may display at least a portion of the images at the fourth refresh rate by changing the refresh rate from the third refresh rate to the fourth refresh rate based on a fourth refresh rate different from the third refresh rate. For example, at least one processor (210) can reduce power consumption within the electronic device (200) by displaying another window (e.g., a window consisting of text and / or images) with a relatively slow state change rate at a relatively low fourth refresh rate. For example, at least one processor (210) can reduce heat generation of the electronic device (200) or increase the battery life of the electronic device (200) by displaying a window (e.g., a window consisting of text and / or images) with a relatively slow state change rate at a relatively low fourth refresh rate.For example, at least one processor (210) may provide the user with another window that changes at a relatively high frequency by displaying another window (e.g., a window consisting of video) with a relatively fast rate of change of state at a relatively high fourth refresh rate. Maintaining or changing the refresh rate depending on whether the user's gaze is positioned over the window for a threshold time is exemplified in the description of FIG. 9.

[0080] Figure 9 illustrates an example of determining the refresh rate based on whether the user's gaze is positioned on the window for a threshold time.

[0081] Referring to FIG. 9, in states (900) and (915), at least one processor (210) may display at least a portion of images at a third refresh rate predetermined in relation to an application providing the window (905), based on whether the user's gaze (910), identified using one or more other cameras (260), is directed toward the window (905). For example, at least one processor (210) may provide augmented reality (410) (or mixed reality, or a non-immersive environment) by displaying at least a portion of the images at the third refresh rate. For example, while displaying at least a portion of the images at the third refresh rate, at least one processor (210) may identify whether the user's (405) gaze (910) is located on the window (905) for a threshold time.

[0082] For example, a state (900) can be described as a state in which the user (405)’s gaze (910) is positioned on the window (905) for a critical time. For example, in the state (900), at least one processor (210) may maintain a refresh rate to display at least a portion of the images at a third refresh rate based on the user (405)’s gaze (910) being positioned on the window (905) for a critical time. For example, as the user (405)’s gaze (910) is positioned on the window (905) for a critical time, at least one processor (210) may determine that the user (405) is focusing on the window (905). For example, as the user (405) is focusing on the window (905), at least one processor (210) may maintain a refresh rate to display at least a portion of the images at a third refresh rate suitable for the window (905).

[0083] For example, the window (905) may be described as a window composed of text and / or images with a relatively slow rate of change in state. For example, as the rate of change in state of the window (905) is relatively slow, there may be no substantial difference between displaying the window (905) at a first refresh rate, which is the maximum refresh rate, and displaying the window (905) at a relatively low third refresh rate. For example, it may be appropriate to display the window (905) with a relatively slow rate of change in state at a relatively low third refresh rate. For example, at least one processor (210) may reduce power consumption within the electronic device (200) by displaying the window (905) with a relatively slow rate of change in state at a relatively low third refresh rate. For example, at least one processor (210) may reduce heat generation of the electronic device (200) or increase the battery life of the electronic device (200) by displaying the window (905) with a relatively slow rate of change in state at a relatively low third refresh rate.

[0084] For example, a state (915) can be described as a state in which the user (405)’s gaze (910) is not located on the window (905) for a critical time. For example, in a state (915), at least one processor (210) can identify another window (925) located on an area adjacent to the window (905) based on the user (405)’s gaze (910) that is not located on the window (905) for a critical time. For example, at least a portion of the other window (925) may be located within a reference distance (920) from the user (405)’s gaze (910) located on the window (905).

[0085] For example, as the user (405)’s gaze (910) is not positioned on the window (905) for a critical time, at least one processor (210) may determine that the user (405) is not focusing on the window (905). For example, as the user (405) is not focusing on the window (905), at least one processor (210) may be required to change a predetermined third refresh rate for the application providing the window (905). For example, as the user (405) is not focusing on the window (905), another window (925) located on an area adjacent to the window (905) may be shown to the user (405). For example, at least one processor (210) may be required to display at least a portion of the images at a fourth refresh rate suitable for the other window (925). For example, the fourth refresh rate may be higher than the third refresh rate. For example, the fourth refresh rate can be described as a relatively high refresh rate corresponding to the first refresh rate or the rate of change of frames according to video playback.

[0086] For example, at least one processor (210) can identify a predetermined fourth refresh rate for an application providing another window (925). For example, the other window (925) may be described as a window composed of video with a relatively fast rate of change in state. For example, as the rate of change in state of the other window (925) is relatively fast, at least some of the changed state of the other window (925) may not be shown to or may be omitted by the user (405) by displaying the other window (925) at a relatively low third refresh rate. For example, it may be appropriate to display the other window (925) with a relatively fast rate of change in state at a relatively high fourth refresh rate. For example, at least one processor (210) may display at least some of the images at the fourth refresh rate by changing the refresh rate from the third refresh rate to the fourth refresh rate. For example, at least one processor (210) can display another window (925) at a fourth refresh rate, thereby providing the content displayed in the other window (925) to the user (405) without omitting it. For example, the operation of changing or maintaining this refresh rate can be performed within an electronic device (200) having the form of glasses. The electronic device (200) having the form of glasses changing or maintaining the refresh rate to display at least a portion of images acquired through one or more cameras (230) is exemplified in the description of FIG. 10.

[0087] FIG. 10 illustrates an example in which an electronic device having the form of glasses displays at least a portion of images.

[0088] Referring to FIG. 10, the state (1000) can be described as a state in which at least a portion of images acquired through one or more cameras (230) is displayed. For example, in the state (1000), an electronic device (200) having the form of glasses may allow a user (405) to view the external environment (or the real world) through a transparent (or translucent) lens. For example, the electronic device (200) having the form of glasses may allow the user (405) to view at least a portion of the images superimposed with the external environment (or the real world) by displaying at least a portion of the images on a transparent (or translucent) lens. For example, at least one processor (210) may provide augmented reality (1005) by displaying at least a portion of the images on a transparent (or translucent) lens through a display assembly (250). For example, an electronic device (200) having the form of glasses can display at least a portion of images on a transparent (or translucent) lens, which can be described as the output of a display assembly (250) that interacts with an external environment (or real world).

[0089] For example, an electronic device (200) having the form of glasses may display at least a portion of images on a transparent (or translucent) lens according to the settings of the user (405) (or according to the brightness of the external environment being lower than the reference brightness). For example, as the external environment is dark, an external object (1010) (e.g., stairs) may be indistinctly seen by the user (405) through the transparent (or translucent) lens of the electronic device (200) having the form of glasses. For example, as the external object (1010) is indistinctly seen by the user (405), the user (405) may collide with the external object (1010) or be in danger. For example, at least one processor (210) may display at least a portion of images on the transparent (or translucent) lens to prevent a collision between the user (405) and the external object (1010). For example, at least one processor (210) can display the outline (1015) of an external object (1010) using at least a portion of the images. For example, the outline (1015) can be displayed in a different color from the external object (1010) so that it can be seen by the user (405) as being distinguishable from the external object (1010). For example, at least one processor (210) can enable the user (405) to clearly see the external object (1010) by displaying the outline (1015) of the external object (1010) using at least a portion of the images.

[0090] For example, an electronic device (200) having the form of glasses can display an outline (1015) on a transparent (or translucent) lens using at least some of the images at a first refresh rate (e.g., 120 Hz), which is the maximum refresh rate. For example, the electronic device (200) having the form of glasses can identify the movement speed of the electronic device (200) having the form of glasses using one or more sensors (240). For example, the electronic device (200) having the form of glasses can maintain the refresh rate to display the outline (1015) at the first refresh rate based on the movement speed of the electronic device (200) having the form of glasses that is higher than the reference speed. For example, the electronic device (200) having the form of glasses can change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate based on the movement speed of the electronic device (200) having the form of glasses that is lower than the reference speed. For example, regarding maintaining or changing the refresh rate according to the movement speed of the electronic device (200) having the form of glasses, the description of FIG. 3 may be referenced.

[0091] For example, an electronic device (200) having the form of glasses can detect an external object moving toward a user (405) using at least some of the images while displaying the outline (1015) at a second refresh rate. For example, the electronic device (200) having the form of glasses can change the refresh rate from the second refresh rate to the first refresh rate based on detecting an external object moving toward a user (405) while displaying the outline (1015) at the second refresh rate. For example, the electronic device (200) having the form of glasses can maintain the refresh rate to display the outline (1015) at the second refresh rate based on identifying that an external object moving toward a user (405) is not detected while displaying the outline (1015) at the second refresh rate. For example, maintaining or changing the refresh rate depending on whether an external object moving toward a user (405) is detected may be described in FIG. 5.

[0092] For example, an electronic device (200) having the form of glasses may display a window on a transparent (or translucent) lens through a display assembly (250) at a second refresh rate while displaying an outline (1015) at a second refresh rate. For example, an electronic device (200) having the form of glasses may identify that the user (405)’s gaze is directed toward the window by using one or more other cameras (260) while displaying the outline (1015) and the window at a second refresh rate. For example, the electronic device (200) having the form of glasses may display the outline (1015) and the window at a third refresh rate determined in relation to the application providing the window, based on the user (405)’s gaze toward the window. For example, the description of FIG. 7 may be referenced for displaying the outline (1015) and the window at a third refresh rate based on the user (405)’s gaze toward the window.

[0093] For example, an electronic device (200) having the form of glasses can identify whether the user's gaze is positioned on the window for a threshold time using one or more other cameras (260) while displaying the outline (1015) and the window at a third refresh rate. For example, the electronic device (200) having the form of glasses can maintain the refresh rate to display the outline (1015) and the window at a third refresh rate based on whether the user's gaze is positioned on the window for a threshold time. For example, the electronic device (200) having the form of glasses can change the refresh rate from the third refresh rate to a fourth refresh rate predetermined in relation to another application providing another window located on an area adjacent to the window, based on whether the user's gaze is not positioned on the window for a threshold time. Refer to the description of FIG. 8 for maintaining or changing the refresh rate depending on whether the user's gaze is positioned on the window for a threshold time.

[0094] FIG. 11 is a block diagram of an electronic device in a network environment according to various embodiments.

[0095] Referring to FIG. 11, in a network environment (1100), an electronic device (1101) may communicate with an electronic device (1102) through a first network (1198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (1104) or a server (1108) through a second network (1199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1101) may communicate with the electronic device (1104) through a server (1108). According to one embodiment, the electronic device (1101) may include a processor (1120), memory (1130), input module (1150), sound output module (1155), display module (1160), audio module (1170), sensor module (1176), interface (1177), connection terminal (1178), haptic module (1179), camera module (1180), power management module (1188), battery (1189), communication module (1190), subscriber identification module (1196), or antenna module (1197). In some embodiments, at least one of these components (e.g., connection terminal (1178)) may be omitted from the electronic device (1101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (1176), camera module (1180), or antenna module (1197)) may be integrated into a single component (e.g., display module (1160)).

[0096] The processor (1120) can, for example, execute software (e.g., program (1140)) to control at least one other component (e.g., hardware or software component) of the electronic device (1101) connected to the processor (1120) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (1120) can store commands or data received from other components (e.g., sensor module (1176) or communication module (1190)) in volatile memory (1132), process the commands or data stored in volatile memory (1132), and store the resulting data in non-volatile memory (1134). According to one embodiment, the processor (1120) may include a main processor (1121) (e.g., a central processing unit or an application processor) or an auxiliary processor (1123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (1101) includes a main processor (1121) and an auxiliary processor (1123), the auxiliary processor (1123) may be configured to use less power than the main processor (1121) or to be specialized for a specified function. The auxiliary processor (1123) may be implemented separately from the main processor (1121) or as part thereof.

[0097] The auxiliary processor (1123) may control at least some of the functions or states associated with at least one component of the electronic device (1101) (e.g., display module (1160), sensor module (1176), or communication module (1190)) on behalf of the main processor (1121) while the main processor (1121) is in an inactive (e.g., sleep) state, or together with the main processor (1121) while the main processor (1121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (1123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (1180) or communication module (1190)). According to one embodiment, the auxiliary processor (1123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (1101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (1108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0098] The memory (1130) can store various data used by at least one component of the electronic device (1101) (e.g., processor (1120) or sensor module (1176)). The data may include, for example, software (e.g., program (1140)) and input or output data for related commands. The memory (1130) may include volatile memory (1132) or non-volatile memory (1134).

[0099] The program (1140) may be stored as software in memory (1130) and may include, for example, an operating system (1142), middleware (1144), or an application (1146).

[0100] The input module (1150) can receive commands or data to be used for a component of the electronic device (1101) (e.g., processor (1120)) from outside the electronic device (1101) (e.g., user). The input module (1150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0101] The sound output module (1155) can output a sound signal to the outside of the electronic device (1101). The sound output module (1155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0102] The display module (1160) can visually provide information to an external (e.g., user) of the electronic device (1101). The display module (1160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (1160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0103] The audio module (1170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (1170) can acquire sound through the input module (1150) or output sound through the sound output module (1155) or an external electronic device (e.g., electronic device (1102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (1101).

[0104] The sensor module (1176) can detect the operating state of the electronic device (1101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (1176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0105] The interface (1177) may support one or more specified protocols that can be used for the electronic device (1101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (1102)). According to one embodiment, the interface (1177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0106] The connection terminal (1178) may include a connector through which the electronic device (1101) can be physically connected to an external electronic device (e.g., electronic device (1102)). According to one embodiment, the connection terminal (1178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0107] The haptic module (1179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (1179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0108] The camera module (1180) can capture still images and video. According to one embodiment, the camera module (1180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0109] The power management module (1188) can manage the power supplied to the electronic device (1101). According to one embodiment, the power management module (1188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0110] The battery (1189) can supply power to at least one component of the electronic device (1101). According to one embodiment, the battery (1189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0111] The communication module (1190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (1101) and an external electronic device (e.g., electronic device (1102), electronic device (1104), or server (1108)), and the performance of communication through the established communication channel. The communication module (1190) may include one or more communication processors that operate independently of the processor (1120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1190) may include a wireless communication module (1192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (1104) through a first network (1198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (1199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1192) can identify or authenticate the electronic device (1101) within a communication network such as the first network (1198) or the second network (1199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (1196).

[0112] The wireless communication module (1192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (1192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (1192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (1192) can support various requirements specified in the electronic device (1101), external electronic device (e.g., electronic device (1104)), or network system (e.g., second network (1199)). According to one embodiment, the wireless communication module (1192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 114 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0113] An antenna module (1197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (1197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (1197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (1198) or a second network (1199), may be selected from the plurality of antennas, for example, by a communication module (1190). A signal or power may be transmitted or received between the communication module (1190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (1197).

[0114] According to various embodiments, the antenna module (1197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0115] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0116] According to one embodiment, commands or data may be transmitted or received between the electronic device (1101) and an external electronic device (1104) through a server (1108) connected to a second network (1199). Each of the external electronic devices (1102, or 1104) may be the same or a different type of device as the electronic device (1101). According to one embodiment, all or part of the operations performed on the electronic device (1101) may be performed on one or more of the external electronic devices (1102, 1104, or 1108). For example, if the electronic device (1101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (1101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (1101). The electronic device (1101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (1101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1104) may include an Internet of Things (IoT) device. The server (1108) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (1104) or server (1108) may be included within the second network (1199). The electronic device (1101) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0117] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0118] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0119] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0120] Various embodiments of the present document may be implemented as software (e.g., program (1140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (1136) or external memory (1138)) readable by a machine (e.g., electronic device (1101)). For example, a processor (e.g., processor (1120)) of the machine (e.g., electronic device (1101)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0121] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0122] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0123] FIG. 12a shows an example of a perspective view of a wearable device.

[0124] FIG. 12b illustrates an example of one or more pieces of hardware placed within a wearable device. According to one embodiment, the electronic device (200) may have the form of glasses that are wearable on a part of a user's body (e.g., head). The electronic device (200) of FIG. 12a and FIG. 12b may be an example of the electronic device (200) of FIG. 2. The electronic device (200) may include a head-mounted display (HMD). For example, the housing of the electronic device (200) may include a flexible material such as rubber and / or silicone that has a shape that adheres to a part of the user's head (e.g., a part of the face covering both eyes). For example, the housing of the electronic device (200) may include one or more straps that can be twined around the user's head and / or one or more temples that are attachable to the ears of the head.

[0125] Referring to FIG. 12a, an electronic device (200) according to one embodiment may include at least one display (1250) and a frame (1200) supporting at least one display (1250).

[0126] According to one embodiment, the electronic device (200) may be worn on a part of a user's body. The electronic device (200) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the electronic device (200). For example, the electronic device (200) may display a virtual reality image provided by at least one optical device (1282, 1284) of FIG. 12b on at least one display (1250) in response to a designated gesture of the user obtained through the motion recognition camera (1260-2, 1260-3) of FIG. 12b.

[0127] According to one embodiment, at least one display (1250) can provide visual information to a user. For example, at least one display (1250) may include a transparent or translucent lens. At least one display (1250) may include a first display (1250-1) and / or a second display (1250-2) spaced apart from the first display (1250-1). For example, the first display (1250-1) and the second display (1250-2) may be positioned at locations corresponding to the user's left eye and right eye, respectively.

[0128] Referring to FIG. 12b, at least one display (1250) may provide visual information transmitted from external light to a user through a lens included in at least one display (1250) and other visual information distinct from said visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (1250) may include a first surface (1231) and a second surface (1232) opposite to the first surface (1231). A display area may be formed on the second surface (1232) of at least one display (1250). When a user wears the electronic device (200), external light may be transmitted to the user by being incident on the first surface (1231) and transmitted through the second surface (1232). As another example, at least one display (1250) can display an augmented reality image combined with a virtual reality image provided by at least one optical device (1282, 1284) on a real screen transmitted through external light in a display area formed on a second surface (1232).

[0129] In one embodiment, at least one display (1250) may include at least one waveguide (1233, 1234) that diffracts light emitted from at least one optical device (1282, 1284) and transmits it to a user. At least one waveguide (1233, 1234) may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed on the exterior or at least a portion of the interior of at least one waveguide (1233, 1234). The nano pattern may be formed based on a polygonal and / or curved grating structure. Light incident on one end of at least one waveguide (1233, 1234) may be propagated to the other end of at least one waveguide (1233, 1234) by the nano pattern. At least one waveguide (1233, 1234) may include at least one diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1233, 1234) may be placed within an electronic device (200) to guide a screen displayed by at least one display (1250) to the user's eye. For example, the screen may be transmitted to the user's eye based on total internal reflection (TIR) ​​occurring within at least one waveguide (1233, 1234).

[0130] The electronic device (200) can analyze an object included in a real-world image collected through a camera (1260-4), combine a virtual object corresponding to an object among the analyzed objects that is the target of augmented reality provision, and display it on at least one display (1250). The virtual object may include at least one of text and an image regarding various information related to the object included in the real-world image. The electronic device (200) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the electronic device (200) can perform spatial recognition (e.g., SLAM (simultaneous localization and mapping)) using a multi-camera and / or time-of-flight (ToF). A user wearing the electronic device (200) can view the image displayed on at least one display (1250).

[0131] According to one embodiment, the frame (1200) may be formed as a physical structure that allows the electronic device (200) to be worn on the user's body. According to one embodiment, the frame (1200) may be configured such that when the user wears the electronic device (200), the first display (1250-1) and the second display (1250-2) can be positioned corresponding to the user's left and right eyes. The frame (1200) may support at least one display (1250). For example, the frame (1200) may support the first display (1250-1) and the second display (1250-2) so that they are positioned corresponding to the user's left and right eyes.

[0132] Referring to FIG. 12a, the frame (1200) may include an area (1220) in which at least a portion of the frame contacts a part of the user's body when the user wears the electronic device (200). For example, the area (1220) of the frame (1200) in contact with a part of the user's body may include an area in contact with a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the electronic device (200) contacts. According to one embodiment, the frame (1200) may include a nose pad (1210) that contacts a part of the user's body. When the electronic device (200) is worn by the user, the nose pad (1210) may contact a part of the user's nose. The frame (1200) may include a first temple (1204) and a second temple (1205) that contact a different part of the user's body distinct from the part of the user's body.

[0133] For example, the frame (1200) may include a first rim (1201) covering at least a portion of a first display (1250-1), a second rim (1202) covering at least a portion of a second display (1250-2), a bridge (1203) positioned between the first rim (1201) and the second rim (1202), a first pad (1211) positioned along a portion of the edge of the first rim (1201) from one end of the bridge (1203), a second pad (1212) positioned along a portion of the edge of the second rim (1202) from the other end of the bridge (1203), a first temple (1204) extending from the first rim (1201) and fixed to a portion of the wearer's ear, and a second temple (1205) extending from the second rim (1202) and fixed to a portion of the ear opposite to the ear. The first pad (1211) and the second pad (1212) may come into contact with a part of the user's nose, and the first temple (1204) and the second temple (1205) may come into contact with a part of the user's face and a part of the ear. The temples (1204, 1205) may be rotatably connected to the rim through the hinge units (1206, 1207) of FIG. 12b. The first temple (1204) may be rotatably connected to the first rim (1201) through a first hinge unit (1206) positioned between the first rim (1201) and the first temple (1204). The second temple (1205) may be rotatably connected to the second rim (1202) through a second hinge unit (1207) disposed between the second rim (1202) and the second temple (1205). According to one embodiment, the electronic device (200) may identify an external object touching the frame (1200) (e.g., a user's fingertip) and / or a gesture performed by said external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame (1200).

[0134] According to one embodiment, the electronic device (200) may include hardware that performs various functions (e.g., hardware to be described later based on the block diagram of FIG. 13). For example, the hardware may include a battery module (1270), an antenna module (1275), at least one optical device (1282, 1284), speakers (e.g., speakers (1255-1, 1255-2)), a microphone (e.g., microphones (1265-1, 1265-2, 1265-3)), a light-emitting module (not shown), and / or a PCB (printed circuit board) (1290) (e.g., a printed circuit board). The various hardware may be placed within a frame (1200).

[0135] According to one embodiment, a microphone (e.g., microphones (1265-1, 1265-2, 1265-3)) of an electronic device (200) is positioned on at least a portion of a frame (1200) to acquire a sound signal. A first microphone (1265-1) positioned on a bridge (1203), a second microphone (1265-2) positioned on a second rim (1202), and a third microphone (1265-3) positioned on a first rim (1201) are shown in FIG. 12b, but the number and position of the microphones (1265) are not limited to the embodiment of FIG. 12b. If there are two or more microphones (1265) included in the electronic device (200), the electronic device (200) can identify the direction of the sound signal by using a plurality of microphones positioned on different portions of the frame (1200).

[0136] According to one embodiment, at least one optical device (1282, 1284) may project a virtual object onto at least one display (1250) to provide various image information to a user. For example, at least one optical device (1282, 1284) may be a projector. At least one optical device (1282, 1284) may be disposed adjacent to at least one display (1250) or included within at least one display (1250) as part of at least one display (1250). According to one embodiment, an electronic device (200) may include a first optical device (1282) corresponding to a first display (1250-1) and a second optical device (1284) corresponding to a second display (1250-2). For example, at least one optical device (1282, 1284) may include a first optical device (1282) positioned at the edge of a first display (1250-1) and a second optical device (1284) positioned at the edge of a second display (1250-2). The first optical device (1282) may transmit light to a first waveguide (1233) positioned on the first display (1250-1), and the second optical device (1284) may transmit light to a second waveguide (1234) positioned on the second display (1250-2).

[0137] In one embodiment, the camera (1260) may include a shooting camera (1260-4), an eye tracking camera (ET CAM) (1260-1), and / or a motion recognition camera (1260-2, 1260-3). The shooting camera (1260-4), the eye tracking camera (1260-1), and the motion recognition camera (1260-2, 1260-3) may be positioned at different locations on the frame (1200) and may perform different functions. The eye tracking camera (1260-1) may output data indicating the position of the eyes or the gaze of a user wearing the electronic device (200). For example, the electronic device (200) may detect the gaze from an image containing the user's pupils obtained through the eye tracking camera (1260-1). The electronic device (200) can identify an object focused by the user (e.g., a real object, and / or a virtual object) by using the user's gaze obtained through the gaze tracking camera (1260-1). The electronic device (200), having identified the focused object, can execute a function for interaction between the user and the focused object (e.g., gaze interaction). The electronic device (200) can represent a portion corresponding to the eyes of an avatar representing the user in a virtual space by using the user's gaze obtained through the gaze tracking camera (1260-1). The electronic device (200) can render an image (or screen) displayed on at least one display (1250) based on the position of the user's eyes. For example, the visual quality of a first region associated with the gaze within the image and the visual quality of a second region distinct from the first region (e.g., resolution, brightness, saturation, grayscale, PPI) may differ from each other. The electronic device (200) can obtain an image having a visual quality of a first region that matches the user's gaze and a visual quality of a second region by using foveated rendering.For example, if the electronic device (200) supports an iris recognition function, user authentication can be performed based on iris information obtained using an eye-tracking camera (1260-1). An example in which the eye-tracking camera (1260-1) is positioned toward the user's right eye is shown in FIG. 12b, but the embodiment is not limited thereto, and the eye-tracking camera (1260-1) may be positioned solely toward the user's left eye or toward both eyes.

[0138] In one embodiment, the camera (1260-4) can capture a real image or background to be matched with a virtual image in order to implement augmented reality or mixed reality content. The camera (1260-4) can be used to acquire high-resolution images based on HR (high resolution) or PV (photo video). The camera (1260-4) can capture an image of a specific object located at the position viewed by the user and provide the image to at least one display (1250). The at least one display (1250) can display a single image in which information regarding a real image or background including the image of the specific object acquired using the camera (1260-4) and a virtual image provided through at least one optical device (1282, 1284) are superimposed. The electronic device (200) can compensate for depth information (e.g., the distance between the electronic device (200) and an external object acquired through a depth sensor) using the image acquired through the camera (1260-4). The electronic device (200) can perform object recognition through an image acquired using a shooting camera (1260-4). The electronic device (200) can perform a function of focusing on an object (or subject) within an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the shooting camera (1260-4). The electronic device (200) can perform a pass-through function to superimpose an image acquired through the shooting camera (1260-4) onto at least a portion of a screen representing a virtual space while displaying the screen representing a virtual space on at least one display (1250). In one embodiment, the shooting camera (1260-4) may be placed on a bridge (1203) positioned between a first rim (1201) and a second rim (1202).

[0139] The eye tracking camera (1260-1) can achieve more realistic augmented reality by tracking the gaze of a user wearing the electronic device (200), thereby matching the user's gaze with visual information provided to at least one display (1250). For example, when the user looks straight ahead, the electronic device (200) can naturally display environmental information related to the user's front on at least one display (1250) at the location where the user is situated. The eye tracking camera (1260-1) may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the eye tracking camera (1260-1) may receive a gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the eye tracking camera (1260-1) may be positioned at locations corresponding to the user's left and right eyes. For example, the eye-tracking camera (1260-1) may be positioned within the first rim (1201) and / or the second rim (1202) to face the direction in which the user wearing the electronic device (200) is located.

[0140] A motion recognition camera (1260-2, 1260-3) can provide a specific event to a screen provided on at least one display (1250) by recognizing the movement of the user's entire body or part thereof, such as the user's torso, hands, or face. A motion recognition camera (1260-2, 1260-3) can recognize the user's gesture, acquire a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display (1250). A processor can identify the signal corresponding to the gesture and, based on the identification, perform a designated function. A motion recognition camera (1260-2, 1260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for a 6-degrees-of-freedom pose (6 dof pose). A processor can use the motion recognition camera (1260-2, 1260-3) to perform a gesture recognition function and / or an object tracking function. In one embodiment, a motion recognition camera (1260-2, 1260-3) may be placed on the first rim (1201) and / or the second rim (1202).

[0141] The camera (1260) included in the electronic device (200) is not limited to the eye-tracking camera (1260-1) and motion recognition camera (1260-2, 1260-3) described above. For example, the electronic device (200) can identify external objects included within the FoV by using a camera positioned toward the user's FoV. The identification of external objects by the electronic device (200) can be performed based on a sensor for identifying the distance between the electronic device (200) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1260) positioned toward the FoV may support an autofocus function and / or an optical image stabilization (OIS) function. For example, the electronic device (200) may include a camera (1260) (e.g., a face tracking camera) positioned toward the face to acquire an image including the face of a user wearing the electronic device (200).

[0142] Although not illustrated, according to one embodiment, the electronic device (200) may further include a light source (e.g., LED) that emits light toward a subject (e.g., user's eye, face, and / or an object outside the FoV) being photographed using a camera (1260). The light source may include an LED of infrared wavelength. The light source may be placed in at least one of the frame (1200) and hinge units (1206, 1207).

[0143] According to one embodiment, the battery module (1270) can supply power to the electronic components of the electronic device (200). In one embodiment, the battery module (1270) may be placed within the first temple (1204) and / or the second temple (1205). For example, the battery module (1270) may be a plurality of battery modules (1270). The plurality of battery modules (1270) may each be placed in the first temple (1204) and the second temple (1205), respectively. In one embodiment, the battery module (1270) may be placed at the end of the first temple (1204) and / or the second temple (1205).

[0144] The antenna module (1275) can transmit a signal or power to the outside of the electronic device (200) or receive a signal or power from the outside. In one embodiment, the antenna module (1275) may be placed within the first temple (1204) and / or the second temple (1205). For example, the antenna module (1275) may be placed close to one side of the first temple (1204) and / or the second temple (1205).

[0145] The speaker (1255) can output an acoustic signal to the outside of the electronic device (200). The acoustic output module may be referred to as the speaker. In one embodiment, the speaker (1255) may be placed within a first temple (1204) and / or a second temple (1205) to be positioned adjacent to the ear of a user wearing the electronic device (200). For example, the speaker (1255) may include a second speaker (1255-2) positioned adjacent to the user's left ear by being placed within the first temple (1204), and a first speaker (1255-1) positioned adjacent to the user's right ear by being placed within the second temple (1205).

[0146] A light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the electronic device (200) to the user. For example, when the electronic device (200) requires charging, it may emit red light at a constant frequency. In one embodiment, the light-emitting module may be placed on the first rim (1201) and / or the second rim (1202).

[0147] Referring to FIG. 12b, according to one embodiment, an electronic device (200) may include a printed circuit board (PCB) (1290). The PCB (1290) may be included in at least one of a first temple (1204) or a second temple (1205). The PCB (1290) may include an interposer disposed between at least two sub-PCBs. On the PCB (1290), one or more hardware components included in the electronic device (200) (e.g., hardware components illustrated by different blocks in FIG. 4) may be disposed. The electronic device (200) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0148] According to one embodiment, the electronic device (200) may include at least one of a gyroscope sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the electronic device (200) and / or the posture of a body part (e.g., head) of a user wearing the electronic device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the electronic device (200) may identify motions and / or gestures of a user performed to execute or interrupt specific functions of the electronic device (200) based on the IMU.

[0149] FIGS. 13a and 13b show examples of the appearance of a wearable device.

[0150] FIGS. 13a and 13b illustrate an example of the appearance of a wearable device (e.g., an electronic device (200)). The electronic device (200) of FIGS. 13a and 13b may be an example of the electronic device (200) of FIG. 2. According to one embodiment, an example of the appearance of a first surface (1310) of the housing of the electronic device (200) may be illustrated in FIG. 13a, and an example of the appearance of a second surface (1320) opposite to the first surface (1310) may be illustrated in FIG. 13b.

[0151] Referring to FIG. 13a, according to one embodiment, a first surface (1310) of an electronic device (200) may have a shape that is attachable to a part of a user's body (e.g., the face of the user). Although not illustrated, the electronic device (200) may further include a strap for fixing to a part of the user's body and / or one or more temples (e.g., a first temple (1204) and / or a second temple (1205) of FIG. 12a to FIG. 12b). A first display (1250-1) for outputting an image to the left eye among the user's two eyes, and a second display (1250-2) for outputting an image to the right eye among the two eyes may be disposed on the first surface (1310). The electronic device (200) may further include a rubber or silicone packing formed on the first surface (1310) to prevent interference by light different from light emitted from the first display (1250-1) and the second display (1250-2) (e.g., ambient light).

[0152] According to one embodiment, the electronic device (200) may include cameras (1260-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (1250-1) and the second display (1250-2). The cameras (1260-1) may be referenced to the eye-tracking camera (1260-1) of FIG. 12b. According to one embodiment, the electronic device (200) may include cameras (1260-5, 1260-6) for photographing and / or recognizing a user's face. The cameras (1260-5, 1260-6) may be referenced to FT cameras. The electronic device (200) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (1260-5, 1260-6). For example, the electronic device (200) can change the texture and / or shape of a part of an avatar (e.g., a part of an avatar representing a human face) by using information obtained by cameras (1260-5, 1260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the electronic device (200).

[0153] Referring to FIG. 13b, on a second surface (1320) opposite to the first surface (1310) of FIG. 13a, a camera (e.g., cameras (1260-7, 1260-8, 1260-9, 1260-10, 1260-11, 1260-12)), and / or a sensor (e.g., a depth sensor (1330)) may be placed to acquire information related to the external environment of the electronic device (200). For example, cameras (1260-7, 1260-8, 1260-9, 1260-10) may be placed on the second surface (1320) to recognize external objects. The cameras (1260-7, 1260-8, 1260-9, 1260-10) can be referenced to the motion recognition cameras (1260-2, 1260-3) of FIG. 12b.

[0154] For example, using cameras (1260-11, 1260-12), the electronic device (200) can acquire images and / or videos to be transmitted to each of the user's two eyes. Camera (1260-11) may be placed on a second surface (1320) of the electronic device (200) to acquire an image to be displayed through a second display (1250-2) corresponding to the right eye among the two eyes. Camera (1260-12) may be placed on a second surface (1320) of the electronic device (200) to acquire an image to be displayed through a first display (1250-1) corresponding to the left eye among the two eyes. Cameras (1260-12, 1260-13) may be referenced to the shooting camera (1260-4) of FIG. 12b.

[0155] According to one embodiment, the electronic device (200) may include a depth sensor (1330) disposed on a second surface (1320) to identify the distance between the electronic device (200) and an external object. Using the depth sensor (1330), the electronic device (200) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the electronic device (200). Although not illustrated, a microphone may be disposed on the second surface (1320) of the electronic device (200) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0156] Hereinafter, with reference to FIG. 14, the hardware or software configuration of the electronic device (200) is described.

[0157] Figure 14 shows an example of a block diagram of a wearable device.

[0158] FIG. 14 illustrates an example of a block diagram of a wearable device (e.g., an electronic device (200)). The electronic device (200) of FIG. 14 may be an example of the electronic device (200) of FIG. 2 and the electronic device (200) of FIG. 12a to FIG. 13b.

[0159] Referring to FIG. 14, an electronic device (200) according to one embodiment may include a processor (1410), memory (1415), a display (1250) (e.g., a first display (1250-1) and / or a second display (1250-2) of FIG. 12a, FIG. 12b, FIG. 13a, and FIG. 13b), and / or a sensor (1414). The processor (1410), memory (1415), display (1250) and / or sensor (1414) may be electrically and / or operationally connected to each other by an electronic component such as a communication bus (1402). In the present disclosure, the operational connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the electronic device (200) are not limited to those shown in FIG. 14. For example, the electronic device (200) may include only some of the electronic components shown in FIG. 14.

[0160] A processor (1410) of an electronic device (200) according to one embodiment may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the electronic device (200) may include one or more processors. The processor (1410) may have a structure of a multi-core processor such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (1410) may include a structure based on multiple core circuits (e.g., a big-little structure), distinguished by power consumption, clock, and / or computational power per unit time. In one embodiment comprising a processor (1410) having a multi-core processor structure, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (1410).

[0161] A memory (1415) of an electronic device (200) according to one embodiment may include electronic components for storing data and / or instructions that are input to or output from a processor (1410). The memory (1415) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multimedia card (eMMC). In one embodiment, the memory (1415) may be referred to as storage.

[0162] In one embodiment, a display (1250) of an electronic device (200) can output visualized information to a user of the electronic device (200). A display (1250) arranged in front of the eyes of a user wearing the electronic device (200) may be placed in at least a part of the housing of the electronic device (200) (e.g., a first display (1250-1) and / or a second display (1250-2) of FIG. 12a, FIG. 12b, FIG. 13a, and FIG. 13b). For example, the display (1250) may be controlled by a processor (1410) including circuits such as a CPU, a GPU (graphic processing unit), and / or a DPU (display processing unit) to output visualized information to the user. The display (1250) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (1250) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, for example, if the electronic device (200) includes a lens for transmitting external light (or ambient light), the display (1250) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (1250) may be referred to as a display panel and / or a display module. Pixels included in the display (1250) may be positioned toward either of the user's two eyes when worn by the user of the electronic device (200).For example, the display (1250) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0163] In one embodiment, a sensor (1414) of an electronic device (200) may generate electrical information that can be processed by a processor (1410) and / or a memory (1415) from non-electronic information related to the electronic device (200). For example, the sensor (1414) may include a global positioning system (GPS) sensor for detecting the geographic location of the electronic device (200). In addition to the GPS method, the sensor (1414) may generate information indicating the geographic location of the electronic device (200) based on a global navigation satellite system (GNSS), such as Galileo or Beidou (compass). This information may be stored in the memory (1415), processed by the processor (1410), and / or transmitted to another electronic device distinct from the electronic device (200) via a communication circuit.

[0164] According to one embodiment, within the memory (1415) of the electronic device (200), one or more instructions (or commands) representing data to be processed by the processor (1410) of the electronic device (200), calculations to be performed, and / or operations may be stored. A set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, sub-routine, and / or software application (hereinafter, application). For example, the electronic device (200), and / or processor (1410) may perform at least one of the operations of FIGS. 3 through 10 when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, program, and / or software application is executed. In the following, the statement that a software application is installed within an electronic device (200) may mean that one or more instructions provided in the form of a software application (or package) are stored in memory (1415), and that said one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the electronic device (200)) by the processor (1410). For example, the application may include a program and / or library related to a service provided to a user.

[0165] Referring to FIG. 14, programs installed on an electronic device (200) may be included in any one of different layers, including an application layer (1440), a framework layer (1450), and / or a hardware abstraction layer (HAL) (1480), based on the target. For example, within the hardware abstraction layer (1480), programs (e.g., modules, or drivers) designed to target the hardware of the electronic device (200) (e.g., a display (1250), and / or a sensor (1414)) may be included. The framework layer (1450) may be referred to as an XR framework layer in that it includes one or more programs for providing XR (extended reality) services. For example, the layers illustrated in FIG. 14 are logically (or for convenience of explanation) separated, and may not imply that the address space of memory (1415) is separated by said layers.

[0166] For example, within the framework layer (1450), programs designed to target at least one of the hardware abstraction layer (1480) and / or the application layer (1440) (e.g., a location tracker (1471), a spatial recognizer (1472), a gesture tracker (1473), an eye tracker (1474), and / or a face tracker (1475)) may be included. The programs included in the framework layer (1450) may provide an application programming interface (API) that is executable (or invokeable) based on other programs.

[0167] For example, within the application layer (1440), a program designed to target a user of the electronic device (200) may be included. Examples of programs included in the application layer (1440) include an XR (extended reality) system UI (user interface) (1441) and / or an XR application (1442), but embodiments are not limited thereto. For example, programs included in the application layer (1440) (e.g., software applications) may call an API to cause the execution of a function supported by programs included in the framework layer (1450).

[0168] For example, the electronic device (200) may display one or more visual objects on the display (1250) to perform interaction with the user based on the execution of the XR system UI (1441). A visual object may mean an object that can be placed on the screen for the transmission of information and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The electronic device (200) may provide the user with functions available in a virtual space based on the execution of the XR system UI (1441).

[0169] Referring to FIG. 14, a lightweight renderer (1443) and / or an XR plugin (1444) are depicted within the XR system UI (1441), but are not limited thereto. For example, based on the XR system UI (1441), the processor (1410) may execute the lightweight renderer (1443) and / or an XR plugin (1444) within the framework layer (1450).

[0170] For example, the electronic device (200) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute a rendering pipeline, which is permitted to be partially modified, based on the execution of a lightweight renderer (1443). The lightweight renderer (1443) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline, which is permitted to be partially modified. The lightweight renderer (1443) may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, the electronic device (200) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute the entire rendering pipeline based on the execution of an XR plugin (1444). The XR plugin (1444) may be referred to as an open XR native client in terms of defining (or setting) the entire rendering pipeline.

[0171] For example, the electronic device (200) may display a screen representing at least a portion of a virtual space on the display (1250) based on the execution of the XR application (1442). The XR plugin (1444-1) included in the XR application (1442) may include instructions that support functions similar to the XR plugin (1444) of the XR system UI (1441). Descriptions of the XR plugin (1444-1) that overlap with descriptions of the XR plugin (1444) may be omitted. The electronic device (200) may trigger the execution of the virtual space manager (1451) based on the execution of the XR application (1442).

[0172] For example, the electronic device (200) may display an image on the display (1250) in virtual space based on the execution of the application (1445). The application (1445) may be configured to output image information for displaying a two-dimensional image. The electronic device (200) may trigger the execution of a virtual space manager (1451) based on the execution of the application (1445). The electronic device (200) may generate dual image information to display the two-dimensional image in three-dimensional virtual space based on the execution of the application (1445). Here, the dual image information may include a first image information for the left eye and a second image information for the right eye, taking into account binocular parallax. To display the two-dimensional image in three-dimensional virtual space, the electronic device (200) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0173] According to one embodiment, the electronic device (200) may provide virtual space services based on the execution of a virtual space manager (1451). For example, the virtual space manager (1451) may include a platform for supporting virtual space services. Based on the execution of the virtual space manager (1451), the electronic device (200) may identify a virtual space formed based on the user's location indicated by data acquired through a sensor (1430), and may display at least a portion of the virtual space on a display (1250). The virtual space manager (1451) may be referred to as a composition presentation manager (CPM).

[0174] For example, the virtual space manager (1451) may include a runtime service (1452). For example, the runtime service (1452) may be referred to as an OpenXR runtime module (or OpenXR runtime program). The electronic device (200) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (1452). For example, the electronic device (200) may perform rendering for a virtual space service for the user based on the execution of the runtime service (1452). For example, a virtual space-related function executable by the application layer (1440) may be supported based on the execution of the runtime service (1452).

[0175] For example, the virtual space manager (1451) may include a pass-through manager (1453). The electronic device (200), based on the execution of the pass-through manager (1453), may display an image and / or video representing the real space acquired through an external camera superimposed on at least a portion of the screen while displaying a screen representing the virtual space on the display (1250).

[0176] For example, the virtual space manager (1451) may include an input manager (1454). The electronic device (200) may identify acquired data (e.g., sensor data) by executing one or more programs included within the recognition service layer (1470) based on the execution of the input manager (1454). The electronic device (200) may identify user inputs associated with the electronic device (200) using the acquired data. The user inputs may be associated with user motions (e.g., hand gestures), gaze, and / or speech identified by a sensor (1414) (e.g., an image sensor (1430) such as an external camera). The user inputs may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0177] For example, the perception abstract layer (1460) can be used for data exchange between the virtual space manager (1451) and the perception service layer (1470). In terms of being used for data exchange between the virtual space manager (1451) and the perception service layer (1470), the perception abstract layer (1460) may be referred to as an interface. As an example, the perception abstract layer (1460) may be referred to as OpenPX. The perception abstract layer (1460) can be used for a perception client and a perception service.

[0178] According to one embodiment, the recognition service layer (1470) may include one or more programs for processing data obtained from the sensor (1414). The one or more programs may include at least one of a location tracker (1471), a spatial recognizer (1472), a gesture tracker (1473), and / or an eye tracker (1474). The type and / or number of the one or more programs included in the recognition service layer (1470) are not limited to those shown in FIG. 14.

[0179] For example, the electronic device (200) can identify the pose of the electronic device (200) using a sensor (1430) based on the execution of a position tracker (1471). The electronic device (200) can identify the 6 degrees of freedom pose (6 dof pose) of the electronic device (200) using data acquired using an external camera (e.g., image sensor (1421)) and / or an IMU (e.g., motion sensor (1422) including a gyroscope, accelerometer, and / or geomagnetic sensor) based on the execution of the position tracker (1471). The position tracker (1471) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0180] For example, the electronic device (200) may acquire information to provide a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the electronic device (200) (or the user of the electronic device (200)) based on the execution of the spatial recognizer (1472). The electronic device (200) may reproduce the surrounding environment of the electronic device (200) in three dimensions using data acquired using an external camera (e.g., image sensor (1421)) based on the execution of the spatial recognizer (1472). The electronic device (200) may identify at least one of a plane, a slope, and a staircase based on the surrounding environment of the electronic device (200) reproduced in three dimensions based on the execution of the spatial recognizer (1472). The spatial recognizer (1472) may be referred to as a scene understanding (SU) module (or scene understanding program).

[0181] For example, the electronic device (200) can identify (or recognize) the pose and / or gesture of the user's hand of the electronic device (200) based on the execution of the gesture tracker (1473). For example, the electronic device (200) can identify the pose and / or gesture of the user's hand using data acquired from an external camera (e.g., image sensor (1421)) based on the execution of the gesture tracker (1473). For example, the electronic device (200) can identify the pose and / or gesture of the user's hand based on data (or images) acquired using an external camera based on the execution of the gesture tracker (1473). The gesture tracker (1473) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0182] For example, the electronic device (200) can identify (or track) the movement of the user's eyes of the electronic device (200) based on the execution of the eye tracker (1474). For example, the electronic device (200) can identify the movement of the user's eyes using data obtained from a gaze tracking camera (e.g., image sensor (1421)) based on the execution of the eye tracker (1474). The eye tracker (1474) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0183] For example, the recognition service layer (1470) of the electronic device (200) may further include a face tracker (1475) for tracking the user's face. For example, the electronic device (200) may identify (or track) the movement of the user's face and / or the user's facial expression based on the execution of the face tracker (1475). The electronic device (200) may estimate the user's facial expression based on the movement of the user's face based on the execution of the face tracker (1475). For example, the electronic device (200) may identify the movement of the user's face and / or the user's facial expression based on data (e.g., images and / or videos) acquired using a camera (1425) (e.g., a camera facing at least a part of the user's face) based on the execution of the face tracker (1475).

[0184] Referring to FIG. 14, the renderer (1490) may include instructions for rendering images in a three-dimensional virtual space. A processor (1410) that executes the renderer (1490) may obtain at least one image to be displayed at least partially in a display area of ​​the display (1250) in a software application. For example, the processor (1410) that executes the renderer (1490) may determine the location of the area where an application (e.g., XR application (1442), application (1445)) will be rendered. The processor (1410) that executes the renderer (1490) may generate an image of said application to be displayed on the display (1250). The renderer (1490) may synthesize images to generate a composite image to be displayed on the display (1250).

[0185] For example, a processor (1410) that executes a renderer (1490) can divide the display area of ​​a display (1250) into a foveated portion (or may be referred to as a foveated area) and a peripheral portion (or may be referred to as a residual area) using a gaze position calculated using a position tracker (1471) and / or a gaze tracker (1474). For example, a processor (1410) that detects coordinate values ​​of the gaze position can determine the portion of the display area containing said coordinate values ​​as the foveated area. A DPU that executes a renderer (1490) can acquire at least one image corresponding to each of said foveated area and said residual area, having a size smaller than the size of the entire display area of ​​the display (1250) or having a resolution less than the resolution of the display area.

[0186] A processor (1410) that executes a renderer (1490) can obtain or generate a composite image to be displayed on a display (1250) by synthesizing an image corresponding to a foveated area and an image corresponding to a surrounding area. For example, the processor (1410) can perform upscaling to enlarge the image corresponding to the surrounding area to the size of the entire display area of ​​the display (1250). On the enlarged image, the processor (1410) can combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (1250). Along the boundary line of the image corresponding to the foveated area, the processor (1410) can mix the enlarged image and the image corresponding to the foveated area by applying a visual effect such as blur.

[0187] Figure 15 shows an example of a block diagram of an electronic device for displaying an image in virtual space.

[0188] In FIG. 15, an example is described in which multiple programs / instructions are executed to display an image in a virtual space. The multiple programs / instructions may all be executed on a single processor (e.g., AP) or may be executed by multiple processors (e.g., AP, GPU (graphic processing unit), NPU (neural processing unit)). The meaning of being able to be executed by multiple processors is that some programs / instructions may be executed by a first processor and other programs / instructions may be executed by a second processor different from the first processor.

[0189] Referring to FIG. 15, the electronic device (200) may execute a virtual space manager (1550) (e.g., the virtual space manager (1451) of FIG. 14, CPM) to render an image in a virtual space. For the virtual space manager (1550), at least some of the descriptions of the virtual space manager (1451) of FIG. 14 may be referenced. The virtual space manager (1550) may include a platform for supporting virtual space services. The virtual space manager (1550) may include a runtime service (1551) (e.g., open XR runtime), a panel renderer (1552) (e.g., 2D panel render), and an XR compositor (1553) (XR compositor). Based on the execution of the runtime service (1551), the electronic device (200) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function. For the runtime service (1551), at least some of the descriptions of the runtime service (1452) of FIG. 14 may be referenced. The electronic device (200) may display at least one image (video) on a panel (e.g., a 2D panel) to enable the implementation of a virtual space through a display, based on the execution of panel rendering (1552). For example, the electronic device (200) may display a rendering image corresponding to RGB information (1566) for a panel from the spatialization manager (1540) described later through a display (e.g., a display (1250)). The electronic device (200) may composite an image of a real area (hereinafter, a pass-through image) captured through a camera in virtual space with an image of a virtual area based on the execution of an XR compositor (1553) (XR compositor). For example, the electronic device (200) can generate a composite image by merging the pass-through image and the virtual region image based on the execution of the XR synthesis unit (1553).The electronic device (200) can transmit the generated composite image to a display buffer so that the composite image is displayed. The electronic device (200) can identify a virtual space through a virtual space manager (1550) and display at least a portion of the virtual space on a display (1250). The virtual space manager (1550) may be referred to as CPM. The electronic device (200) can execute the virtual space manager (1550) to render an image corresponding to at least a portion of the virtual space.

[0190] According to one embodiment, the electronic device (200) may execute a spatialization manager (1540). The spatialization manager (1540) may perform processing for displaying an image in a three-dimensional virtual space. The electronic device (200) may perform preprocessing based on the execution of the spatialization manager (1540) so that an image can be rendered in a three-dimensional virtual space through a virtual space manager (1550). For example, the electronic device (200) may perform at least some of the functions of the renderer (1490) of FIG. 14 based on the execution of the spatialization manager (1540). The electronic device (200) may process image information provided by an application (e.g., an XR application (1510), an application providing a general 2D screen that is not XR (1520), an application providing a system UI (1530)) based on the execution of the spatialization manager (1540). A spatialization manager (1540) (e.g., space flinger) may include a system screen manager (1541) (e.g., system scene), an input manager (1542) (e.g., input routing), and a lightweight rendering engine (1543) (e.g., impress engine). The system screen manager (1541) may be executed to display a system UI (1530). System UI-related information (1564) may be transmitted to the system screen manager (1541) from a program (e.g., API) that provides the system UI (1530). System UI-related information (1564) may be obtained through a spatializer API and / or a same-process private API. The spatialization manager (1540) may determine the layout (e.g., position, display order) of the system UI (1530) screen in three-dimensional space through pre-allocated resources.The system screen manager (1541) may transmit image information (1567) for rendering a screen of the system UI (1530) according to the layout to the virtual space manager (1550). The input manager (1542) may be configured to process user input (e.g., user input on a system screen or app screen). The impression engine (1543) may be a renderer for image generation (e.g., a lightweight renderer (1543)). For example, the impression engine (1543) may be used to display the system UI (1530). According to one embodiment, the spatialization manager (1540) may include a lightweight rendering engine (1543) for rendering the system UI. According to one embodiment, if the lightweight rendering engine (1543) does not have sufficient resources to render an avatar used in an HMD, at least one external rendering engine may be used. At this time, to resolve compatibility issues with external rendering (e.g., 3rd party engine), an external rendering engine support module may be added inside the spatialization manager (1540).

[0191] According to one embodiment, the electronic device may execute an application. For example, in response to the execution of an XR application (1510) (e.g., XR application (1510), 3D game, XR map, other immersive application), the virtual space manager (1550) may be executed. The electronic device (200) may provide dual image information (1561) provided from the XR application (1510) to the virtual space manager (1550). To display images in three-dimensional space, the dual image information (1561) may include two image information that takes into account binocular parallax. For example, the dual image information (1561) may include a first image information for the user's left eye and a second image information for the user's right eye to render in three-dimensional virtual space. Hereinafter, the term dual image information is used in the present disclosure to refer to image information for displaying images for both eyes in three-dimensional space. In addition to the dual image information, the above dual image information may utilize binocular image information, dual image information, dual image data, dual image, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimension conversion image data, binocular parallax image data, and / or equivalent technical terms. The electronic device (200) can generate a composite image by merging image layers through a virtual space manager (1550). The electronic device (200) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (1250) of the electronic device (200).

[0192] According to one embodiment, the electronic device may execute at least one application among an XR application (1510) and other applications (1520) (e.g., a first application (1520-1), a second application (1520-2), ..., a Nth application (1520-N)). According to one embodiment, the application (1520) may be configured to output image information for displaying a two-dimensional image. In other words, the application (1520) may provide a two-dimensional image. As an example, the application (1520) may be a video application, a schedule application, or an internet browser application. Let us assume that, in response to the execution of the application (1520), image information (1562) provided from the application (1520) is provided to a virtual space manager (1550). Since the image information (1562) has only x and y coordinates within a two-dimensional plane, it may be difficult to consider the sequential relationship between other applications centered on the user (i.e., distance from the user). The electronic device (200) may execute a spatialization manager (1540) to provide dual image information to a virtual space manager (1550) even when displaying an application (1520) that provides a general 2D screen. For example, based on the execution of the spatialization manager (1540), the electronic device (200) may receive application-related information (1563) from the first application (1520-1). For example, the application-related information (1563) may include image information representing a two-dimensional image of the first application (1520-1) (e.g., information including RGB per pixel) and / or content information in the first application (1520-1) (e.g., characteristics of the content executed in the first application, type of content). Application-related information (1563) can be obtained through the spatializer API.Based on the execution of the spatialization manager (1540), the electronic device (200) can identify information (hereinafter, location information) regarding the location of the area to be rendered and the size of the area to be rendered. Based on the execution of the spatialization manager (1540), the electronic device (200) can generate dual image information (1565, e.g., RGBx2) that takes into account the user's binocular parallax through the image information and the location information. Based on the execution of the spatialization manager (1540), the electronic device (200) can provide the dual image information (1565) to the virtual space manager (1550). By converting a simple two-dimensional image into dual image information (1565), the problem caused by the image information (1562) being directly transmitted to the virtual space manager (1550) can be resolved. Additionally, as at least some of the functions for displaying images in virtual space are performed by the spatialization manager (1540) instead of the virtual space manager (1550), the burden on the virtual space manager (1550) may be reduced.

[0193] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0194] As described above, the electronic device worn on the head of the user (e.g., the electronic device (200) of FIG. 2) may include at least one processor (e.g., at least one processor (210) of FIG. 2) including a processing circuit, one or more cameras (e.g., one or more cameras (230) of FIG. 2), one or more sensors (e.g., one or more sensors (240) of FIG. 2), a display assembly (e.g., the display assembly (250) of FIG. 2), and a memory (e.g., memory (220) of FIG. 2) including one or more storage media configured to store one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the electronic device to acquire images corresponding to the user's field of view (FOV) through the one or more cameras. The above one or more programs may include instructions that cause the electronic device to display at least a portion of the images on the display assembly at a first refresh rate. The above one or more programs may include instructions that cause the electronic device to identify the movement speed of the electronic device (e.g., movement speed (415) of FIG. 4) using the above one or more sensors. The above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the first refresh rate based on the movement speed which is greater than the reference speed while displaying at least a portion of the images at the first refresh rate.The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying, during a reference time, a movement speed lower than the reference speed identified while displaying at least a portion of the images at the first refresh rate. The above one or more programs may include instructions that cause the electronic device to change the refresh rate back to the first refresh rate, based on detecting an external object (e.g., external object (605) of FIG. 6) moving toward the user using at least a portion of the images while displaying at least a portion of the images at the second refresh rate.

[0195] For example, the above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the second refresh rate, based on identifying that the external object moving toward the user is not detected while displaying at least a portion of the images at the second refresh rate.

[0196] For example, the electronic device may further include one or more other cameras (e.g., one or more other cameras (260) of FIG. 2). The one or more programs may include instructions that cause the electronic device to identify that the user's gaze (e.g., gaze (910) of FIG. 9), identified using other images acquired through the one or more other cameras while maintaining the refresh rate to display at least a portion of the images at the second refresh rate, is directed toward a window (e.g., window (905) of FIG. 9) on at least a portion of the images displayed at the second refresh rate. The one or more programs may include instructions that cause the electronic device to identify a third refresh rate predetermined for the application providing the window based on identifying the gaze directed toward the window. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate at the second refresh rate based on the third refresh rate corresponding to the second refresh rate. The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the second refresh rate to the third refresh rate based on the third refresh rate which is different from the second refresh rate.

[0197] For example, the one or more programs may include instructions that cause the electronic device to identify whether the gaze is positioned on the window for a threshold time while displaying at least a portion of the images at the third refresh rate. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate to display at least a portion of the images at the third refresh rate based on the gaze positioned on the window for the threshold time.

[0198] For example, the one or more programs may include instructions that cause the electronic device to identify another window located on an area adjacent to the window based on the line of sight that is not located on the window for the threshold time. The one or more programs may include instructions that cause the electronic device to identify a predetermined fourth refresh rate for another application providing the other window. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate at the third refresh rate based on the fourth refresh rate corresponding to the third refresh rate. The one or more programs may include instructions that cause the electronic device to change the refresh rate from the third refresh rate to the fourth refresh rate based on the fourth refresh rate different from the third refresh rate.

[0199] For example, at least a portion of the other window may be located within a reference distance from the area where the line of sight is located on the window.

[0200] For example, the third refresh rate may be predetermined as a refresh rate lower than the first refresh rate with respect to the application providing text content and / or image content. The third refresh rate may be predetermined as a refresh rate corresponding to the first refresh rate with respect to the application providing video content.

[0201] The above-described method may be performed within an electronic device worn on a user's head, comprising one or more cameras, one or more sensors, and a display assembly. The method may include the operation of acquiring images corresponding to the user's field of view (FOV) through the one or more cameras. The method may include the operation of displaying at least a portion of the images on the display assembly at a first refresh rate. The method may include the operation of identifying the movement speed of the electronic device using the one or more sensors. The method may include the operation of maintaining the refresh rate to display at least a portion of the images at the first refresh rate based on the movement speed greater than the reference speed while displaying at least a portion of the images at the first refresh rate. The method may include the operation of changing the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate based on identifying the movement speed lower than the reference speed identified while displaying at least a portion of the images at the first refresh rate for a reference time. The above method may include an operation of changing the refresh rate back to the first refresh rate based on detecting an external object moving toward the user using at least a portion of the images while displaying at least a portion of the images at the second refresh rate.

[0202] For example, the above method may include an operation to maintain a refresh rate to display at least a portion of the images at the second refresh rate, based on identifying that the external object moving toward the user is not detected while displaying at least a portion of the images at the second refresh rate.

[0203] For example, the electronic device may further include one or more other cameras. The method may include an operation of identifying that the user’s gaze, identified using other images acquired through the one or more other cameras while maintaining a refresh rate to display at least a portion of the images at the second refresh rate, is directed toward a window on at least a portion of the images displayed at the second refresh rate. The method may include an operation of identifying a third refresh rate predetermined for an application providing the window based on identifying the gaze directed toward the window. The method may include an operation of maintaining the refresh rate at the second refresh rate based on the third refresh rate corresponding to the second refresh rate. The method may include an operation of changing the refresh rate from the second refresh rate to the third refresh rate based on the third refresh rate different from the second refresh rate.

[0204] For example, the method may include an operation of identifying whether the gaze is positioned on the window for a threshold time while at least a portion of the images is displayed at the third refresh rate. The method may include an operation of maintaining the refresh rate to display at least a portion of the images at the third refresh rate based on the gaze positioned on the window for the threshold time.

[0205] For example, the method may include an operation of identifying another window located on an area adjacent to the window based on the line of sight that is not located on the window for the threshold time. The method may include an operation of identifying a predetermined fourth refresh rate for another application providing the other window. The method may include an operation of maintaining the refresh rate at the third refresh rate based on the fourth refresh rate corresponding to the third refresh rate. The method may include an operation of changing the refresh rate from the third refresh rate to the fourth refresh rate based on the fourth refresh rate which is different from the third refresh rate.

[0206] For example, at least a portion of the other window may be located within a reference distance from the area where the line of sight is located on the window.

[0207] For example, the third refresh rate may be predetermined as a refresh rate lower than the first refresh rate with respect to the application providing text content and / or image content. The third refresh rate may be predetermined as a refresh rate corresponding to the first refresh rate with respect to the application providing video content.

[0208] The above-described non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the electronic device to acquire images corresponding to the user's field of view (FOV) through the one or more cameras when executed by the electronic device worn on the user's head, the electronic device comprising one or more cameras, one or more sensors, and a display assembly. The one or more programs may include instructions that cause the electronic device to display at least a portion of the images on the display assembly at a first refresh rate when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify the movement speed of the electronic device using the one or more sensors when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the first refresh rate, based on a moving speed greater than a reference speed while displaying at least a portion of the images at the first refresh rate when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying, during a reference time, a moving speed lower than the reference speed identified while displaying at least a portion of the images at the first refresh rate when executed by the electronic device.The above one or more programs may include instructions that cause the electronic device to change the refresh rate back to the first refresh rate based on detecting an external object moving toward the user using at least a portion of the images while displaying at least a portion of the images at the second refresh rate when executed by the electronic device.

[0209] For example, the above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the second refresh rate, based on identifying that, when executed by the electronic device, at least a portion of the images is displayed at the second refresh rate while the external object moving toward the user is not detected.

[0210] For example, the electronic device may further include one or more other cameras. The one or more programs may include instructions that cause the electronic device to identify that the user’s gaze, identified using other images acquired through the one or more other cameras, is directed toward a window on at least a portion of the images displayed at the second refresh rate while maintaining a refresh rate to display at least a portion of the images at the second refresh rate when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify a predetermined third refresh rate for an application providing the window based on identifying the gaze directed toward the window when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate at the second refresh rate based on the third refresh rate corresponding to the second refresh rate when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the second refresh rate to the third refresh rate based on the third refresh rate which is different from the second refresh rate when executed by the electronic device.

[0211] For example, the one or more programs may include instructions that cause the electronic device to identify whether the gaze is positioned on the window for a threshold time while at least a portion of the images is displayed at the third refresh rate when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate to display at least a portion of the images at the third refresh rate based on the gaze positioned on the window for the threshold time when executed by the electronic device.

[0212] For example, the one or more programs may include instructions that cause the electronic device to identify another window located on an area adjacent to the window based on the line of sight that is not located on the window for the threshold time when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify a predetermined fourth refresh rate for another application providing the other window when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate at the third refresh rate based on the fourth refresh rate corresponding to the third refresh rate when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to change the refresh rate from the third refresh rate to the fourth refresh rate based on the fourth refresh rate different from the third refresh rate when executed by the electronic device.

[0213] For example, at least a portion of the other window may be located within a reference distance from the area where the line of sight is located on the window.

[0214] For example, the third refresh rate may be predetermined as a refresh rate lower than the first refresh rate with respect to the application providing text content and / or image content. The third refresh rate may be predetermined as a refresh rate corresponding to the first refresh rate with respect to the application providing video content.

[0215] The electronic device worn on the head of the user as described above may include at least one processor including a processing circuit, one or more cameras, one or more sensors, a display assembly, and a memory including one or more storage media configured to store one or more programs configured to be executed individually or collectively by the at least one processor. The one or more programs may include instructions that cause the electronic device to acquire images corresponding to the user's field of view (FOV) through the one or more cameras. The one or more programs may include instructions that cause the electronic device to display at least a portion of the images on the display assembly at a first refresh rate. The one or more programs may include instructions that cause the electronic device to identify the movement speed of the electronic device using the one or more sensors, and to identify whether an external object moving toward the user is detected using at least a portion of the images. The above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the first refresh rate, based on identifying the movement speed greater than the reference speed while displaying at least a portion of the images at the first refresh rate, or detecting the external object while displaying at least a portion of the images at the first refresh rate.The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying the movement speed lower than the reference speed during a reference time while displaying at least a portion of the images at the first refresh rate.

[0216] For example, the above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the second refresh rate, based on identifying that the external object moving toward the user is not detected while displaying at least a portion of the images at the second refresh rate.

[0217] For example, the electronic device may further include one or more other cameras. The one or more programs may include instructions that cause the electronic device to identify that the user’s gaze, identified using other images acquired through the one or more other cameras while maintaining a refresh rate to display at least a portion of the images at the second refresh rate, is directed toward a window on at least a portion of the images displayed at the second refresh rate. The one or more programs may include instructions that cause the electronic device to identify a third refresh rate predetermined for an application providing the window based on identifying the gaze directed toward the window. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate at the second refresh rate based on the third refresh rate corresponding to the second refresh rate. The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the second refresh rate to the third refresh rate based on the third refresh rate which is different from the second refresh rate.

[0218] For example, the one or more programs may include instructions that cause the electronic device to identify whether the gaze is positioned on the window for a threshold time while displaying at least a portion of the images at the third refresh rate. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate to display at least a portion of the images at the third refresh rate based on the gaze positioned on the window for the threshold time.

[0219] For example, the one or more programs may include instructions that cause the electronic device to identify another window located on an area adjacent to the window based on the line of sight that is not located on the window for the threshold time. The one or more programs may include instructions that cause the electronic device to identify a predetermined fourth refresh rate for another application providing the other window. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate at the third refresh rate based on the fourth refresh rate corresponding to the third refresh rate. The one or more programs may include instructions that cause the electronic device to change the refresh rate from the third refresh rate to the fourth refresh rate based on the fourth refresh rate different from the third refresh rate.

[0220] For example, at least a portion of the other window may be located within a reference distance from the area where the line of sight is located on the window.

[0221] For example, the third refresh rate may be predetermined as a refresh rate lower than the first refresh rate with respect to the application providing text content and / or image content. The third refresh rate may be predetermined as a refresh rate corresponding to the first refresh rate with respect to the application providing video content.

[0222] The above-described method may be performed within an electronic device worn on a user's head, comprising one or more cameras, one or more sensors, and a display assembly. The method may include the operation of acquiring images corresponding to the user's field of view (FOV) through the one or more cameras. The method may include the operation of displaying at least a portion of the images on the display assembly at a first refresh rate. The method may include the operation of identifying the movement speed of the electronic device using the one or more sensors, and identifying whether an external object moving toward the user is detected using at least a portion of the images. The method may include the operation of maintaining the refresh rate to display at least a portion of the images at the first refresh rate, based on identifying the movement speed greater than a reference speed while displaying at least a portion of the images at the first refresh rate, or detecting the external object while displaying at least a portion of the images at the first refresh rate. The above method may include an operation of changing the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying the movement speed lower than the reference speed during a reference time while displaying at least a portion of the images at the first refresh rate.

[0223] For example, the above method may include an operation to maintain a refresh rate to display at least a portion of the images at the second refresh rate, based on identifying that the external object moving toward the user is not detected while displaying at least a portion of the images at the second refresh rate.

[0224] For example, the electronic device may further include one or more other cameras. The method may include an operation of identifying that the user’s gaze, identified using other images acquired through the one or more other cameras while maintaining a refresh rate to display at least a portion of the images at the second refresh rate, is directed toward a window on at least a portion of the images displayed at the second refresh rate. The method may include an operation of identifying a third refresh rate predetermined for an application providing the window based on identifying the gaze directed toward the window. The method may include an operation of maintaining the refresh rate at the second refresh rate based on the third refresh rate corresponding to the second refresh rate. The method may include an operation of changing the refresh rate from the second refresh rate to the third refresh rate based on the third refresh rate different from the second refresh rate.

[0225] For example, the method may include an operation of identifying whether the gaze is positioned on the window for a threshold time while at least a portion of the images is displayed at the third refresh rate. The method may include an operation of maintaining the refresh rate to display at least a portion of the images at the third refresh rate based on the gaze positioned on the window for the threshold time.

[0226] For example, the method may include an operation of identifying another window located on an area adjacent to the window based on the line of sight that is not located on the window for the threshold time. The method may include an operation of identifying a predetermined fourth refresh rate for another application providing the other window. The method may include an operation of maintaining the refresh rate at the third refresh rate based on the fourth refresh rate corresponding to the third refresh rate. The method may include an operation of changing the refresh rate from the third refresh rate to the fourth refresh rate based on the fourth refresh rate which is different from the third refresh rate.

[0227] For example, at least a portion of the other window may be located within a reference distance from the area where the line of sight is located on the window.

[0228] For example, the third refresh rate may be predetermined as a refresh rate lower than the first refresh rate with respect to the application providing text content and / or image content. The third refresh rate may be predetermined as a refresh rate corresponding to the first refresh rate with respect to the application providing video content.

[0229] The above-described non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the electronic device to acquire images corresponding to the user's field of view (FOV) through the one or more cameras when executed by an electronic device worn on the user's head, the electronic device comprising one or more cameras, one or more sensors, and a display assembly. The one or more programs may include instructions that cause the electronic device to display at least a portion of the images on the display assembly at a first refresh rate. The one or more programs may include instructions that cause the electronic device to identify the movement speed of the electronic device using the one or more sensors and identify whether an external object moving toward the user is detected using at least a portion of the images when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the first refresh rate, based on identifying a moving speed greater than a reference speed while displaying at least a portion of the images at the first refresh rate, or detecting an external object while displaying at least a portion of the images at the first refresh rate, when executed by the electronic device.The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying the movement speed lower than the reference speed during a reference time while displaying at least a portion of the images at the first refresh rate when executed by the electronic device.

[0230] For example, the above one or more programs may include instructions that cause the electronic device to maintain a refresh rate to display at least a portion of the images at the second refresh rate, based on identifying that, when executed by the electronic device, at least a portion of the images is displayed at the second refresh rate while the external object moving toward the user is not detected.

[0231] For example, the electronic device may further include one or more other cameras. The one or more programs may include instructions that cause the electronic device to identify that the user’s gaze, identified using other images acquired through the one or more other cameras, is directed toward a window on at least a portion of the images displayed at the second refresh rate while maintaining a refresh rate to display at least a portion of the images at the second refresh rate when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify a predetermined third refresh rate for an application providing the window based on identifying the gaze directed toward the window when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate at the second refresh rate based on the third refresh rate corresponding to the second refresh rate when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to change the refresh rate from the second refresh rate to the third refresh rate based on the third refresh rate which is different from the second refresh rate when executed by the electronic device.

[0232] For example, the one or more programs may include instructions that cause the electronic device to identify whether the gaze is positioned on the window for a threshold time while at least a portion of the images is displayed at the third refresh rate when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate to display at least a portion of the images at the third refresh rate based on the gaze positioned on the window for the threshold time when executed by the electronic device.

[0233] For example, the one or more programs may include instructions that cause the electronic device to identify another window located on an area adjacent to the window based on the line of sight that is not located on the window for the threshold time when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify a predetermined fourth refresh rate for another application providing the other window when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to maintain the refresh rate at the third refresh rate based on the fourth refresh rate corresponding to the third refresh rate when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to change the refresh rate from the third refresh rate to the fourth refresh rate based on the fourth refresh rate different from the third refresh rate when executed by the electronic device.

[0234] For example, at least a portion of the other window may be located within a reference distance from the area where the line of sight is located on the window.

[0235] For example, the third refresh rate may be predetermined as a refresh rate lower than the first refresh rate with respect to the application providing text content and / or image content. The third refresh rate may be predetermined as a refresh rate corresponding to the first refresh rate with respect to the application providing video content.

[0236] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

Claims

In an electronic device (200) worn on the user's head, Memory that stores instructions and includes one or more storage media; One or more cameras (230); One or more sensors (240); Display assembly (250); and It includes at least one processor comprising a processing circuit, and When the above instructions are executed individually or collectively by the at least one processor, Images corresponding to the user's field of view (FOV) are obtained through the one or more cameras (230), and At least some of the images are displayed on the display assembly (250) at a first refresh rate, and Using the above one or more sensors (240), the movement speed (415) of the electronic device (200) is identified, and While displaying at least a portion of the images at the first refresh rate, based on the movement speed (415) which is greater than the reference speed, the refresh rate is maintained to display at least a portion of the images at the first refresh rate, and Based on identifying the movement speed (415) lower than the reference speed during a reference time while displaying at least a portion of the images at the first refresh rate, the refresh rate is changed from the first refresh rate to a second refresh rate lower than the first refresh rate, and While displaying at least a portion of the images at the second refresh rate, the refresh rate is changed back from the second refresh rate to the first refresh rate based on detecting an external object (605) moving toward the user using at least a portion of the images. causing the above electronic device (200), Electronic device (200). In claim 1, When the above instructions are executed individually or collectively by the at least one processor, Based on identifying that the external object (605) moving toward the user is not detected while displaying at least a portion of the images at the second refresh rate, the refresh rate is maintained to display at least a portion of the images at the second refresh rate. causing the above electronic device (200), Electronic device (200). In claim 2, It further includes one or more other cameras (260), and When the above instructions are executed individually or collectively by the at least one processor, While maintaining the refresh rate to display at least a portion of the images at the second refresh rate, identifying that the user's gaze (910), identified using other images acquired through the one or more other cameras (260), is directed toward a window (905) on at least a portion of the images displayed at the second refresh rate, Based on identifying the line of sight (910) toward the window (905), a predetermined third refresh rate is identified for the application providing the window (905), and Based on the third refresh rate corresponding to the second refresh rate, the refresh rate is maintained at the second refresh rate, and Based on the third refresh rate which is different from the second refresh rate, the refresh rate is changed from the second refresh rate to the third refresh rate. causing the above electronic device (200), Electronic device (200). In claim 3, When the above instructions are executed individually or collectively by the at least one processor, Identifying whether the line of sight (910) is positioned on the window (905) for a threshold time while displaying at least some of the images at the third refresh rate, and Based on the line of sight (910) located on the window (905) during the threshold time, to maintain the refresh rate to display at least a portion of the images at the third refresh rate, causing the above electronic device (200), Electronic device (200). In claim 4, When the above instructions are executed individually or collectively by the at least one processor, Based on the line of sight (910) that is not located on the window (905) during the threshold time, another window (925) located on an area adjacent to the window (905) is identified, and Identifying a predetermined fourth refresh rate for another application that provides the above other window (925), and Based on the fourth refresh rate corresponding to the third refresh rate, the refresh rate is maintained at the third refresh rate, and Based on the fourth refresh rate which is different from the third refresh rate, the refresh rate is changed from the third refresh rate to the fourth refresh rate. causing the above electronic device (200), Electronic device (200). In claim 5, At least some of the other windows (925) mentioned above are, The above line of sight (910) is located within a reference distance from the area located on the window (905), Electronic device (200). In claim 3, The above third regeneration rate is, With respect to the application providing text content and / or image content, a refresh rate lower than the first refresh rate is predetermined, and Regarding the above application providing video content, a playback rate predetermined as a playback rate corresponding to the first playback rate, Electronic device (200). In an electronic device (200) worn on the user's head, Memory that stores instructions and includes one or more storage media; One or more cameras (230); One or more sensors (240); Display assembly (250); and It includes at least one processor comprising a processing circuit, and When the above instructions are executed individually or collectively by the at least one processor, Images corresponding to the user's field of view (FOV) are obtained through the one or more cameras (230), and At least some of the images are displayed on the display assembly (250) at a first refresh rate, and Using one or more of the above sensors (240), the movement speed (415) of the electronic device (200) is identified, and using at least some of the above images, whether an external object (605) moving toward the user is detected is identified. While displaying at least a portion of the images at the first refresh rate, based on identifying the movement speed (415) which is greater than the reference speed, or detecting the external object (605) while displaying at least a portion of the images at the first refresh rate, maintaining the refresh rate to display at least a portion of the images at the first refresh rate, and Based on identifying the movement speed (415) lower than the reference speed during a reference time while displaying at least a portion of the images at the first refresh rate, the refresh rate is changed from the first refresh rate to a second refresh rate lower than the first refresh rate. causing the above electronic device (200), Electronic device (200). In claim 8, When the above instructions are executed individually or collectively by the at least one processor, Based on identifying that the external object (605) moving toward the user is not detected while displaying at least a portion of the images at the second refresh rate, the refresh rate is maintained to display at least a portion of the images at the second refresh rate. causing the above electronic device (200), Electronic device (200). In claim 9, It further includes one or more other cameras (260), and When the above instructions are executed individually or collectively by the at least one processor, While maintaining the refresh rate to display at least a portion of the images at the second refresh rate, identifying that the user's gaze (910), identified using other images acquired through the one or more other cameras (260), is directed toward a window (905) on at least a portion of the images displayed at the second refresh rate, Based on identifying the line of sight (910) toward the window (905), a predetermined third refresh rate is identified for the application providing the window (905), and Based on the third refresh rate corresponding to the second refresh rate, the refresh rate is maintained at the second refresh rate, and Based on the third refresh rate which is different from the second refresh rate, the refresh rate is changed from the second refresh rate to the third refresh rate. causing the above electronic device (200), Electronic device (200). In claim 10, When the above instructions are executed individually or collectively by the at least one processor, Identifying whether the line of sight (910) is positioned on the window (905) for a threshold time while displaying at least some of the images at the third refresh rate, and Based on the line of sight (910) located on the window (905) during the threshold time, to maintain the refresh rate to display at least a portion of the images at the third refresh rate, causing the above electronic device (200), Electronic device (200). In claim 11, When the above instructions are executed individually or collectively by the at least one processor, Based on the line of sight (910) that is not located on the window (905) during the threshold time, another window (925) located on an area adjacent to the window (905) is identified, and Identifying a predetermined fourth refresh rate for another application that provides the above other window (925), and Based on the fourth refresh rate corresponding to the third refresh rate, the refresh rate is maintained at the third refresh rate, and Based on the fourth refresh rate which is different from the third refresh rate, the refresh rate is changed from the third refresh rate to the fourth refresh rate. causing the above electronic device (200), Electronic device (200). In claim 12, At least some of the other windows (925) mentioned above are, The above line of sight (910) is located within a reference distance from the area located on the window (905), Electronic device (200). In claim 10, The above third regeneration rate is, With respect to the application providing text content and / or image content, a refresh rate lower than the first refresh rate is predetermined, and Regarding the above application providing video content, a playback rate predetermined as a playback rate corresponding to the first playback rate, Electronic device (200). A method of execution within an electronic device (200) worn on a user's head, comprising one or more cameras (230), one or more sensors (240), and a display assembly (250), wherein the method is The operation of acquiring images corresponding to the user's field of view (FOV) through the one or more cameras (230), The operation of displaying at least a portion of the images on the display assembly (250) at a first refresh rate, An operation to identify the movement speed (415) of the electronic device (200) using the above one or more sensors (240), The operation of maintaining a refresh rate to display at least a portion of the images at the first refresh rate based on the moving speed (415) which is greater than the reference speed while displaying at least a portion of the images at the first refresh rate, The operation of changing the refresh rate from the first refresh rate to a second refresh rate lower than the first refresh rate, based on identifying the movement speed (415) lower than the reference speed during a reference time while displaying at least a portion of the images at the first refresh rate, and While displaying at least a portion of the images at the second refresh rate, the operation of changing the refresh rate back from the second refresh rate to the first refresh rate based on detecting an external object (605) moving toward the user using at least a portion of the images, method.

Images