Postpone state changes affecting information of a graphical user interface until periods of inattention

Abstract

The present disclosure relates to postponing state changes affecting information of a graphical user interface until a period of inattention. Triggering a state change includes configuring a display to display a first version of a series of frames based on a first settings configuration; obtaining a second settings configuration for the series of frames; in response to obtaining the second settings configuration, monitoring for a change in an eye state; and in response to detecting the change in the eye state, configuring the display to display a second version of the series of frames based on the second settings configuration.

Description

[0001] This application is a divisional application of PCT International Application No. 201780058417.5, filed on September 22, 2017, entitled "Postponing State Changes Affecting a Graphical User Interface Until a Non-Attentive Condition," which entered the Chinese national phase as of September 22, 2017.

[0002] Cross Reference to Related Applications

[0003] This patent application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 62 / 398,438, filed on September 22, 2016, the contents of which are incorporated by reference in their entirety. TECHNICAL FIELD

[0004] The present disclosure relates generally to the field of digital image processing, and more specifically to the field of graphical user interface (GUI) design. In particular, it relates to postponing state changes affecting a GUI and triggering state changes during a condition of non-attentiveness of a user of the GUI. BACKGROUND

[0005] Current technology allows users to interact with their environment in a variety of ways. By way of example, from GPS devices to gaming software, users are allowed to interact with real-world environments in innovative ways. One problem with mixing virtual information into a real-world environment is that changes to the graphical user interface can be sudden and distracting to the user, thereby compromising the user experience.

[0006] For example, a state change in information on a GUI can change while the user is focused on it, causing confusion, or can change while the user is focused on something nearby in the GUI, which can cause the user to be distracted. Words or objects in a computer game can be dynamically loaded and appear suddenly. The level of detail displayed on a screen can change suddenly, causing the user to be confused. There is a need for enhanced user experiences to handle state changes in the data being displayed. SUMMARY

[0007] In one embodiment, a method for triggering a state change is described. The method can include displaying a first version of a series of frames based on a first setting configuration, obtaining a second setting configuration for the series of frames, monitoring a change in an eye state in response to obtaining the second setting configuration, and displaying a second version of the series of frames based on the second setting configuration in response to detecting the change in the eye state. In another embodiment, the method can be embodied in computer executable program code and stored in a non-transitory storage device. In another embodiment, the method can be implemented in an electronic device having image capture capabilities. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 A simplified electronic device according to one or more embodiments is shown in block diagram form.

[0009] Figure 2 A method for managing state changes affecting a GUI according to one or more embodiments is shown in flowchart form.

[0010] Figure 3 Another method for managing state changes affecting a GUI in accordance with one or more embodiments is illustrated in flowchart form.

[0011] Figure 4 An example flow diagram illustrating state changes in managing levels of detail according to one or more embodiments is shown.

[0012] Figure 5 An example flow diagram of a method for managing state changes based on updated keyframes is shown in accordance with one or more embodiments.

[0013] Figure 6 An example flow diagram of a method for managing state changes based on an updated level of detail is shown in accordance with one or more embodiments.

[0014] Figure 7 An example flow diagram of a method for managing state changes based on an updated level of detail is shown in accordance with one or more embodiments.

[0015] Figure 8 An example system diagram of a computing device is shown in accordance with one or more embodiments. DETAILED DESCRIPTION

[0016] The present disclosure relates to systems, methods, and computer-readable media for deferring state changes affecting information of a graphical user interface (GUI) during conditions of inattention. Generally speaking, techniques for managing a display on a GUI are disclosed such that when a set configuration of a series of frames is detected, a new version of the series of frames can be rendered, but not displayed until a user is determined to be inattentive. For example, according to one or more embodiments, a first version of the series of frames can be displayed until it is determined that the user has blinked or looked away. According to one or more embodiments, changing the display during a state of inattention by the user can minimize or reduce the distraction of the change to the user.

[0017] According to one or more embodiments, upon detecting a new settings configuration, the system can render a second version of the series of frames while displaying the first series of frames. That is, the first version of the series of frames and the second version of the series of frames can be rendered in parallel. In some embodiments, both versions of the series of frames can be rendered until the second version of the frames can be displayed. The second series of frames can be delayed until a change in eye state is detected or anticipated. Thus, the two versions of the series of frames can be generated in parallel within a shorter period of time.

[0018] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the disclosed concepts. As part of this specification, some of the drawings of the present disclosure represent structures and devices in block diagram form to avoid blurring the novel aspects of the disclosed embodiments. In this context, it should be understood that references to numbered drawing elements without associated identifiers (e.g., 100) refer to all instances of drawing elements with identifiers (e.g., 100a and 100b). In addition, as part of this specification, some of the drawings of the present disclosure may be provided in the form of flow charts. The blocks in the flow charts may be presented in a specific order. However, it should be understood that the specific processes of any flow chart are only used to illustrate one embodiment. In other embodiments, any of the various components depicted in the flow charts may be deleted, or the components may be executed in a different order, or even simultaneously. In addition, other embodiments may include additional steps that are not shown as part of the flow charts. The language used in this disclosure is primarily selected for readability and guidance purposes, and may not be selected to depict or limit the disclosed subject matter. Reference in this disclosure to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same embodiment.

[0019] It will be appreciated that in the development of any actual implementation (as in any development project), numerous decisions must be made to achieve the developer's specific goals (e.g., conforming to system and business-related constraints), and that these goals will vary from one implementation to another. It will also be appreciated that such development work may be complex and time-consuming, but will nonetheless be a routine undertaking for those of ordinary skill in the art who benefit from the image capture disclosed herein.

[0020] For the purposes of this disclosure, the term "camera" refers to a single lens assembly as well as a sensor element and other circuitry for capturing an image. For the purposes of this disclosure, two or more cameras may share a single sensor element and other circuitry, but include two different lens assemblies. However, in one or more embodiments, the two or more cameras may include independent sensor elements and separate lens assemblies and circuitry.

[0021] See also Figure 1 , which shows a simplified block diagram of an electronic device 100 according to one or more embodiments of the present disclosure. The electronic device 100 can be part of a multi-function device, such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, or any other electronic device that includes a camera system, a display, and other components for implementing various embodiments. The electronic device 100 can be connected to other network devices, such as mobile devices, tablet devices, desktop devices, and network storage devices such as servers, via a network.

[0022] The electronic device 100 may include a central processing unit (CPU) 130. The processor 130 may be a system-on-chip such as those found in mobile devices and may include one or more dedicated graphics processing units (GPUs). In addition, the processor 130 may include multiple processors of the same or different types. The electronic device 100 may also include a memory 140. The memory 140 may include one or more different types of memory that can be used to perform device functions in conjunction with the CPU 130. For example, the memory 140 may include a cache, ROM, and / or RAM. The memory 140 may store various programming modules during execution, including an eye state monitor 155 and a graphics display module 160. According to one or more embodiments, the memory 140 may include additional applications for generating and managing a graphical user interface based on dynamic setting configuration data. For example, the memory 140 may additionally include a rendering engine.

[0023] The electronic device 100 may also include one or more cameras, such as a front-facing camera 110 and a rear-facing camera 120. Cameras 110 and 120 may each include an image sensor, a lens stack, and other components useful for capturing images. In one or more embodiments, the cameras may be oriented in different directions within the electronic device. For example, the front-facing camera 110 may be located in or on a first surface of the electronic device 100, while the rear-facing camera 120 may be located in or on a second surface of the electronic device 100. In one or more embodiments, the first and second surfaces may be opposing surfaces of the electronic device 100. For example, the front-facing camera 110 may be configured to capture images of the real world from the user's perspective, while the rear-facing camera 120 may be configured to capture images of the user. Although two cameras are shown, in some embodiments, as described in further detail below, the electronic device 100 may include a single camera. In other embodiments, the electronic device 100 may also include a display 150. The sensor 175 may include any type of sensor useful for, for example, determining eye state, such as gaze direction or whether the eyes are open or closed. The display 150 may be any type of display device, such as an LCD display, an LED display, an OLED display, or the like. In addition, the display 150 may be a semi-opaque display, such as a heads-up display. Although the electronic device 100 is described as including the numerous components described above, in one or more embodiments, various components may be distributed across multiple devices. Furthermore, additional components may be used, and some combination of the functionality of any component may be combined.

[0024] According to one or more embodiments, the eye state monitor 155 monitors the state of the eyes. In some embodiments, the shape of the eyes will indicate whether the user is concentrating or not. That is, the eye state monitor 155 can determine whether the user is concentrating based on data received, for example, from the front camera 110 or from various sensors 175. In addition, the eye state monitor 155 can anticipate changes in the state of the eyes. For example, the image from the front camera 110 may indicate that the eyes are closed or are closing, or whether the eyes are looking in a direction different from the display 150. For another example, a sensor that monitors the muscles around the eyes can detect that the eyes are about to blink, thereby predicting a change in the state of the eyes. In one or more embodiments, the eye state monitor 155 can continuously monitor the state of the eyes, or can monitor the state of the eyes periodically or when needed. For example, the eye state monitor 155 can monitor the state of the eyes in response to determining that a new setting configuration is available.

[0025] According to one or more embodiments, the graphics display module 160 may be configured to generate a series of frames based on a given setup configuration. In one or more embodiments, the setup configuration may be based on data received locally, for example, via sensor 175 or cameras 110 and 120. Additionally or alternatively, data for the setup configuration may be received from a remote source, such as a network device or server. For example, if the electronic device 100 is displaying directions on a map, the setup configuration may include data from a GPS unit (not shown). The graphics display module 160 may occasionally receive updated setup configuration information, which changes the GUI's view to a series of frames. For example, if updated GPS information indicates that the user is actually on a different road or lane than indicated by the initial setup configuration data, a second version of the series of frames may be rendered. In one or more embodiments, the graphics display module 160 may generate multiple series of frames simultaneously. Returning to this embodiment, if updated GPS data is received, the graphics display module 160 may begin rendering the second version of the series of frames while still rendering and displaying the first series of frames. Thus, the graphics display module 160 may switch from the first version of the series of frames to the second version of the series of frames as needed. According to one or more embodiments, the graphics display module 160 may determine when to begin displaying the second version of the series of frames based on the change in eye state determined by the eye state monitor 155. In some embodiments, when it is determined that the user is inattentive, the graphics display module 160 may switch the version of the configuration information displayed. In other words, the graphics display module 160 may postpone the change in the state of the graphical representation in the GUI until the inattention condition is met.

[0026] For example, in a computer game, a new setting configuration may include new objects that become displayable in the game environment. Graphics display module 160 may continue to present the game environment without the new objects, and present them when the user blinks, so that the new objects are displayed after the blink. Similarly, in augmented reality applications, graphics display module 160 may postpone displaying changes to rendered virtual objects in the real environment. For another example, a progressive JPEG may transition from coarser blocks to finer blocks, or vice versa. Graphics display module 160 may postpone displaying the updated level of detail until the user is distracted. This setting configuration may also be applied to the level of detail in other ways. For example, when zooming a map, information such as street names or neighborhoods may appear or disappear. Graphics display module 160 may postpone changing the level of detail displayed on the map until the user is distracted.

[0027] Figure 2A method for managing state changes affecting a GUI in accordance with one or more embodiments is shown in the form of a flow diagram. Although various actions are depicted as occurring in a specific order, in different embodiments various actions can be performed in a different order. Moreover, in some embodiments, two or more actions can be performed concurrently or with partial concurrence. In other embodiments, some actions can not be performed. For the sake of clarity, the flow diagram will be described with respect to the Figure 1 components of the system 100 of FIG. 1. However, it should be understood that various actions could be performed by alternative components.

[0028] The flow diagram begins at 205, where the graphical display module 160 causes a first version of a series of frames to be displayed on the display 150. In one or more embodiments, the graphical display module 160 can generate the initial series of frames based on a first settings configuration. The first settings configuration can be any data that indicates how a series of frames should be rendered. In one or more other embodiments, the graphical display module 160 can present a series of frames captured by a camera such as the rear-facing camera 120, or can present frames generated by another method. In addition, the series of frames can include additional information rendered based on other data. For example, data collected from a remote server or from the sensors 175. For example, the graphical display module 160 can present virtual objects in a view of a real environment captured by the rear-facing camera 120 on the display 150.

[0029] The flow diagram continues at 210, and determines whether an update in the settings configuration is identified. As described above, an update in the settings configuration can be any type of data that determines how a series of frames is rendered. For example, more or less detail can be used, or depending on other factors, it can be preferable to include a greater or lesser level of detail. If no update is detected, the flow diagram continues at 205, and the graphical display module 160 continues to display the first version of the series of frames based on the first settings configuration. Returning to 210, if it is determined that an update in the settings configuration is identified, the flow diagram continues at 215, and the graphical display module 160 presents a second version of the series of frames based on the updated settings configuration.

[0030] The flowchart continues at 220 where the eye state monitor 155 monitors changes in eye state. According to one or more embodiments, the eye state monitor 155 may monitor changes in gaze or blinking, either of which may indicate an inattentive state of the user. In some embodiments, the eye state monitor 155 may monitor an image of the eye, for example, provided by the front-facing camera 110. Additionally, in other embodiments, the eye state monitor 155 may rely on additional sensor data from the sensor 175 to determine changes in eye state. According to some embodiments, the eye state monitor 155 may continuously monitor the state of the user's eye, or may begin monitoring the state of the eye in response to identification of a settings configuration update or in response to the graphics display module 160 rendering a second version of the series of frames based on the updated settings configuration.

[0031] The flowchart continues at 225 where it is determined whether a change in eye state is detected. According to one or more embodiments, a change in eye state may indicate that the user is distracted. For example, the user is blinking, or looking outside the display screen. In other embodiments, if one or both of the user's eyes are still open, looking outside may not be enough to distract the user. Instead, the user must look in such a way that the user is not aware of the change in the display. For example, if the user can still notice a sudden change in the graphics of the GUI, simply looking at a different location may not be enough. For example, in the user's peripheral vision, if no change in eye state is detected at 225, the flowchart continues at 220, where the eye state monitor 155 continues to monitor for changes in eye state. Returning to 225, if it is determined that a change in eye state is detected, the flowchart continues at 230, where the graphics display module 160 begins displaying a second version of the series of frames based on the second setting configuration.

[0032] Figure 3 Another method for managing state changes affecting a GUI in accordance with one or more embodiments is illustrated in flowchart form. Figure 3 The flow chart can be shown Figure 2 Although various actions are depicted in a particular order, in some embodiments, the various actions may be depicted in a different order. Additionally, in other embodiments, two or more actions may occur simultaneously. Furthermore, according to other embodiments, some actions may not be required, or other actions may be included. For clarity, the description of Figure 1 However, it should be understood that, according to one or more embodiments, various actions may be performed by alternative components.

[0033] The flowchart begins at 305, where the graphics display module 160 causes a first version of a series of frames to be displayed on the display 150. In one or more embodiments, the graphics display module 160 may generate the initial series of frames based on a setup configuration. The first setup configuration may be any data that indicates how the series of frames should be presented. In other embodiments, the graphics display module 160 may present a series of frames captured by a camera, such as the rear-facing camera 120, or may present frames generated by another method. Furthermore, the series of frames may include additional information presented based on other data, such as data collected from a remote server or from sensors 175. For example, the graphics display module 160 may present a virtual object on the display 150 within the view of the real environment captured by the rear-facing camera 120.

[0034] The flowchart continues at 310 where it is determined whether an update in the settings configuration is identified. As described above, an update in the settings configuration may be any type of data that determines how the series of frames should be presented. For example, more or less detail may be used, or depending on other factors, it may be preferable to include a greater or lesser level of detail. If no update is detected, the flowchart continues at 305 where the graphics display module 160 continues to display the first version of the series of frames based on the first settings configuration. However, Figure 2 If, contrary to the steps depicted in , an update in the settings configuration is identified at 310 , the flowchart continues at 315 , where the graphics display module 160 may continue to display the first version of the series of frames.

[0035] The flowchart continues at 320 where the eye state monitor 155 monitors the eye to determine if a change in eye state is predicted. According to one or more embodiments, the change in eye state may be predicted based on an image from the front-facing camera 110, for example. In other embodiments, the image may indicate, for example, that the eye is closing. As another example, other sensor data may be collected from the sensor 175 to determine if a blink or other movement is occurring that would indicate an expected change in eye state. For example, the sensor data may monitor muscle movement of muscles surrounding the eye to predict when a blink is about to occur. In response to determining that a change in eye state is expected, the flowchart continues at 325 where the graphics display module 160 renders a second version of the series of frames based on the updated settings configuration.

[0036] The flowchart continues at 330, where the eye state monitor 155 monitors for changes in eye state. According to one or more embodiments, the eye state monitor 155 may monitor the eyes to verify changes in eye state. For example, the eye state monitor 155 may monitor to verify blinking, rather than just anticipating blinking. The flowchart continues at 335, where it is determined whether a change in eye state is detected. According to some embodiments, a change in eye state may indicate that the user is inattentive. For example, the user is blinking or looking away from the display. In one or more other embodiments, if one or both of the user's eyes are still open, looking away may not be sufficient to determine that the user is inattentive. Instead, the user must look in such a way that the user is not aware of the changes in the display 150. For example, if the user can still notice a sudden change in the graphics of the GUI (e.g., in the user's peripheral vision), simply looking to a different location may not be sufficient. If a change in eye state is not detected at 335, the flowchart continues at 330, where the eye state monitor 155 continues to monitor for changes in eye state. Returning to 335 , if it is determined that a change in eye state is detected, the flowchart continues at 340 , where the graphics display module 160 begins displaying a second version of the series of frames based on a second setting configuration.

[0037] According to one or more embodiments, Figure 2 Flowchart and Figure 3 The difference between the flowcharts of is when to start rendering the second version of the frame. In one or more embodiments, it may be preferred to start rendering the second version of the frame when the updated setting configuration is available, such as Figure 2 , so as to quickly change between the display of the first and second versions of the frame. Furthermore, according to other embodiments, it may be preferable to begin rendering the second version of the frame in response to an anticipated change in eye state. For example, it may be preferable to defer rendering the second version of the frame in order to conserve resources. According to still other embodiments, the time at which graphics display module 160 begins rendering the second version of the frame may be determined based on other factors, such as whether computing resources should be conserved or a configured rate of change.

[0038] Figure 4An exemplary use case is shown in which a change in level of detail is postponed until the user loses focus. Initially, the level of detail used to render a virtual object may depend on the distance of the virtual object from the camera. Each level of detail, and for example, the associated 3D model, may be associated with a specific distance interval. When the distance exceeds a certain threshold, a change in the level of detail used may be triggered. For example, a change in level of detail may be triggered when the distance leaves one interval and enters another interval. In 401, a sphere object is close to the camera, causing it to be rendered using a high-resolution model. When the model moves away from the camera (as in 402), the same level of detail may be used. However, at some point in time (421), the object is so far away that another level of detail is selected. In response to exceeding the threshold distance at 421, the sphere object may be rendered with less detail, as shown in 413. The change from image 402 to image 413 will be obvious to the user. However, at 422, this change in level of detail is again postponed until the user loses focus. As a result, the user will continue to view the high-resolution model in frames 403 and 404. In one or more embodiments, frame 415 executes and displays the change to the low-resolution network only when the user is distracted at 422 .

[0039] Figure 5 Another example use case is shown where the method is used in an augmented reality system where a virtual tank (551) is rendered into a camera image (512-516) of a real world environment. Figure 5 In one or more embodiments, state changes are delayed relative to selected keyframes in the pose determination algorithm. Thus, in one or more embodiments, a sudden change in the appearance (such as position, scale, or orientation) of the virtual tank in the image is delayed so that the user does not notice the change.

[0040] For the purposes of this embodiment, the virtual object is a tank (551), the setup configuration is a specific keyframe (e.g., 502) used to determine the camera pose for the captured image, and the visualization configuration is a specific determined pose of the camera / object used to render the virtual object. The visualization frame is an augmented image (e.g., 522) showing a camera image (e.g., 512) of the real-world environment overlaid with a rendering of the virtual object (551).

[0041] The real-world environment may include a scene of a real table (e.g., (541)). The model of the scene may be used as keyframes, each keyframe consisting of an image of the table (e.g., 502, 503) and a corresponding known 6-DOF camera pose, called a keypose. Figure 5 In the embodiment of FIG, the key frame images (502, 503) show the table (541, 542) from different viewpoints.

[0042] A mobile camera can then be used to capture the scene. For example, the camera of a mobile device. Figure 5 The second column (511) depicts a sequence of captured images, such as real-time video frames (511). For each captured image, a pose determination method can determine the camera pose based on the captured image itself, a model of the scene, and possibly a history of previous captured images and poses.

[0043] For a first captured image (512) of the real world, the algorithm can find the best fitting keyframe 502 and determine the camera pose for image (512) based on image (512), keyframe image (502), and the known key pose of keyframe image (502). The camera pose of image (512) based on keyframe image (502) can be used to render the virtual can (551) on top of image (512), resulting in augmented image (522).

[0044] For a second captured image (513) of the real world, the algorithm can continue to consider the same keyframe image (502) as the best fitting keyframe image. For example, the keyframe image can be determined to be the best fitting keyframe image, or the algorithm to determine the best keyframe can not be triggered for each frame, and the algorithm that considered the best for the previous frame is used. Thus, the algorithm can determine the pose for image (513) based on image (513), keyframe image (502), and the known key pose of keyframe image (502). Alternatively, the algorithm can determine the pose for image (513) based on image (513), image (512), and the previously determined pose for image (512), which was determined based on keyframe image (502) and the known key pose of keyframe image (502). In both cases, the determined pose can initially depend on keyframe (502). The camera pose for image (513) determined based on keyframe (502) can be used to render the virtual can (552) on top of image (513), resulting in augmented image (523).

[0045] For the third acquired image (514) of the real-world environment, the algorithm can now determine that another keyframe image is the best-fit image, namely keyframe (503). The standard method can now directly use the newly determined best-fit keyframe (503) to determine the pose of the image (514) based on the image (514), the keyframe image (503), and the known key pose of the keyframe image (503). The camera pose for the image (514) determined based on the keyframe (503) can be used to render the virtual tank (555) on top of the image (514), thereby producing an augmented image (534). This change in the keyframe used (i.e., the setup configuration) can potentially result in a visible, sudden change in the pose (visualization configuration) and, therefore, in the rendering of the virtual object, which will be noticed by the user. However, if the user is not paying attention, the new keyframe (503) can be used to determine the pose, and the final augmented image can be image (534). However, in the embodiment shown, the user is focused, so keyframe (502) is still used to determine the pose for the visualization, and the final augmented image is image (524). It should be noted that due to the imprecision of the tracking algorithm and the pose determined from keyframe (502), the can (553) may not be rendered flush with the top of the table but may be slightly suspended above the table. That is, keyframe (503) may provide a more accurate pose visible in image (534), where the can (555) is better aligned with the table. It is also possible to determine a pose from the new keyframe (503) to enable frame-to-frame tracking. However, this pose may not be used for the current rendering of the virtual object.

[0046] For the fourth acquired image of the real world (515), the algorithm may again consider the keyframe image (503) to be the best fitting keyframe image. This may be because the algorithm for determining the best keyframe may not be triggered for each frame and use the one deemed best from the previous frame. If the user is not paying attention, the new keyframe (503) may be used to determine the pose (either directly or based on tracking frame to frame based on the previous pose initially based on keyframe 503), and the final augmented image will be image (535). However, in the illustrated embodiment, the user is again paying attention, so the keyframe (502) may be used to determine the pose, and the final augmented image is image (525). Similarly, due to the inaccuracy of the tracking algorithm and the pose determined from keyframe (502), the can (554) may not be rendered flush with the top of the table, but rather may be floating above the table, or intersecting the table. In one or more embodiments, the keyframe (503) may potentially deliver a more accurate pose, as in image (535), where the can (556) is better aligned with the table.

[0047] Figure 6Another exemplary use case is shown where the user or the application itself zooms into a city map. When initially zoomed out (601), most of the city is displayed and only coarse information is presented (major streets (606), city name (607)). When zoomed in at 602, only major streets (608) and city name (609) continue to be displayed. However, when zoomed in further, the application may attempt to display an image, such as 604, with additional information (fewer major streets (613), labels for points of interest (611), underground transportation (614)). Instead, the city name (612) may no longer be displayed. When the additional information 619 is displayed may depend on availability, such as when dynamic streaming content is complete or on the specification of keeping the GUI from being cluttered with too many labels / layers.

[0048] Likewise, the proposed method may maintain the old settings configuration (only the main streets and city name (610) visible) for rendering, resulting in image 603. Changing to the "new" settings configuration (e.g., visibility of smaller streets) is postponed until the user loses focus (620). After detecting that the user is losing focus, the "new" / second settings configuration is used for rendering, resulting in image 605 with smaller streets (616), labels for points of interest (615), underground transportation (618) now visible, and the city name (617) no longer displayed. Since the user is losing focus when the settings configuration (visibility of layers) changes, he / she is likely to miss the change.

[0049] Figure 7Another example use case is shown where the user pans on a city map, or the GPS location is updated and the application centers the map accordingly. In the initial display image (701) of the map, streets (707) and labels for points of interest (706) are displayed. By scrolling the map down, new parts of the map become visible in the next displayed image (702). These new parts (708) initially do not have any labels (e.g., "Residenz" is missing). Missing labels can be caused, for example, by dynamic content streaming or by specifications to avoid clipped labels. When the user scrolls down at some point in time (714) (due to complete download or due to specifications), new labels for the new map areas may suddenly appear, as shown in image 704, where the label "Residenz" (709) appears. However, the old settings configuration used for rendering results in image 703, where the label "Residenz" is still not visible (710). Changing to the "new" settings configuration (with new labels, e.g., Residenz visible) may be postponed until the user loses focus. When user inattention is detected (715), the "new" settings configuration is available for rendering, resulting in image 705 with new labels now visible (711, 712, 713). Since the user is inattentive when the settings configuration (visibility of labels) changes, he / she is likely to miss the change.

[0050] refer to Figure 8 , which shows a simplified functional block diagram of an exemplary multifunction electronic device 800 according to one embodiment. The multifunction electronic device 800 may include a processor 805, a display 810, a user interface 815, graphics hardware 820, device sensors 825 (e.g., proximity sensor / ambient light sensor, accelerometer, and / or gyroscope), a microphone 830, an audio codec 835, a speaker 840, communication circuitry 845, digital image capture circuitry 850 (e.g., including camera system 100), a video codec 855 (e.g., supporting digital image capture unit 850), a memory 860, a storage device 865, and a communication bus 870. The multifunction electronic device 800 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0051] The processor 805 may execute instructions necessary to perform or control the operation of many functions performed by the device 800 (e.g., such as the generation and / or processing of images and single-camera and multi-camera calibration disclosed herein). The processor 805 may, for example, drive the display 810 and may receive user input from the user interface 815. The user interface 815 may allow a user to interact with the device 800. For example, the user interface 815 may take various forms such as buttons, a keypad, a dial, a click wheel, a keyboard, a display screen, and / or a touch screen. The processor 805 may also be, for example, a system on a chip, such as those found in mobile devices and including a dedicated graphics processing unit (GPU). The processor 805 may be based on a reduced instruction set computer (RISC) architecture or a complex instruction set computer (CISC) architecture or any other suitable architecture and may include one or more processing cores. The graphics hardware 820 may be dedicated computing hardware for processing graphics and / or assisting the processor 805 in processing graphics information. In one embodiment, the graphics hardware 820 may include a programmable GPU.

[0052] The image capture circuitry 850 may include one (or more) lens assemblies 880, each of which may have a separate focal length. Each lens assembly may have a separate associated sensor element 890. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuitry 850 may capture still images and / or video images. The output from the image capture circuitry 850 may be processed, at least in part, by one or more video codecs 865 and / or processor 805 and / or graphics hardware 820, and / or a dedicated image processing unit or pipeline incorporated within the circuitry 850. Thus, the captured images may be stored in memory 860 and / or storage device 855.

[0053] The sensor and camera circuitry 850 may capture still and video images that may be processed, at least in part, in accordance with the present disclosure by the video codec 855 and / or the processor 805 and / or the graphics hardware 820, and / or a dedicated image processing unit incorporated within the circuitry 850. Thus, the captured images may be stored in memory 860 and / or storage 865. Memory 860 may include one or more different types of media used by the processor 805 and the graphics hardware 820 to perform device functions. For example, memory 860 may include a memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage 865 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 865 may include one or more non-transitory storage media, including, for example, magnetic disks (fixed disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video disks (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs). The memory 860 and the storage device 865 may be used to tangibly hold computer program instructions or codes organized into one or more modules and written in any desired computer programming language. For example, when executed by the processor 805, such computer program code may implement one or more of the methods described herein.

[0054] The scope of the subject matter disclosed herein should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein."

Claims

1. A method for triggering a state change, comprising: displaying a first series of frames, wherein the first series of frames are rendered based on image data and a first setting configuration; obtaining a second setup configuration for the image data during display of the first series of frames; responsive to obtaining the second setting configuration, monitoring an eye state of an eye of the user to detect an anticipated inattention state; responsive to detecting an anticipated inattention state, rendering a second series of frames based on the image data and the second settings configuration concurrently with the rendering of at least the first portion of the first series of frames while displaying at least the second portion of the first series of frames; as well as In response to detecting the inattentive state, the second series of frames is caused to be displayed on the display device.

2. The method according to claim 1, wherein Monitoring the eye status includes: capturing sensor data corresponding to the user's eyes; and Based on the sensor data, it is determined that the user is about to enter the inattentive state.

3. The method of claim 2, wherein detecting the inattention state comprises monitoring a gaze direction of the eyes to confirm that the eyes are not gazing at the display device. The method of claim 2 , wherein detecting an inattentive state comprises monitoring the sensor data for positive blinks.

5. The method of claim 4, wherein monitoring the sensor data to confirm a positive blink further comprises: acquiring a plurality of image frames from a camera facing an eye corresponding to the eye state, and The plurality of image frames are monitored for blinks of the eye. The method of claim 1 , wherein the first setting configuration and the second setting configuration are associated with virtual content displayed in a real environment.

7. The method of claim 1 , wherein the first setup configuration is associated with a first keyframe for rendering a virtual object based on a first camera pose, and wherein the second setup configuration is obtained in response to determining that the second keyframe is an improved match for an updated camera pose.

8. A non-transitory computer readable medium comprising computer readable code for executing the method according to any one of claims 1 to 7.

9. A system for triggering a state change, comprising: Display devices; one or more processors; and One or more computer-readable media comprising computer-readable code executable by the one or more processors to: displaying a first series of frames, wherein the first series of frames are rendered based on image data and a first setting configuration; obtaining a second setup configuration for the image data during display of the first series of frames; responsive to obtaining the second setting configuration, monitoring an eye state of an eye of the user to detect an anticipated inattention state; responsive to detecting an anticipated inattention state, rendering a second series of frames based on the image data and the second settings configuration concurrently with the rendering of at least the first portion of the first series of frames while displaying at least the second portion of the first series of frames; as well as In response to detecting the inattentive state, the second series of frames is displayed on a display device.

10. The system according to claim 9, wherein: The computer readable code for monitoring the eye state includes computer readable code for: capturing sensor data corresponding to the user's eyes; and Based on the sensor data, it is determined that the user is about to enter the inattentive state.

11. The system of claim 10, wherein the computer readable code for detecting an inattentive state comprises computer readable code for monitoring a gaze direction of an eye to confirm that the eye is not gazing at a display device.

12. The system of claim 10, wherein the computer readable code for detecting an inattentive state comprises computer readable code for monitoring the sensor data for positive blinks.

13. The system of claim 12, wherein the computer readable code that monitors the sensor data to confirm a positive blink further comprises computer readable code for: acquiring a plurality of image frames from a camera facing an eye corresponding to the eye state, and The plurality of image frames are monitored for blinks of the eye.

14. The system of claim 13, wherein the first setting configuration and the second setting configuration are associated with virtual content displayed in a real environment.

15. A method for triggering a state change, comprising: rendering a first series of frames for presentation to a user, the first series of frames comprising virtual content at a first level of detail; obtaining additional details for the virtual content; monitoring an eye state of the user's eyes to detect an anticipated inattention state based on obtaining the additional details for the virtual content; responsive to detecting the expected inattention state, rendering a second series of frames, the second series of frames including the virtual content with the additional detail, wherein the second series of frames are rendered during presentation of the first series of frames; as well as The determination that the state of the user's eyes is indicative of an inattentive state of the user causes the presentation of the first series of frames to be replaced with the second series of frames.

16. The method of claim 15, wherein monitoring eye status comprises: capturing sensor data corresponding to the user's eyes; as well as Based on the sensor data, it is determined that the user is about to enter the inattentive state.

17. The method of claim 16, wherein detecting an inattentive state comprises monitoring a gaze direction of the eye to confirm that the eye is not gazing at a display device.

18. The method of claim 16, wherein detecting an inattentive state comprises monitoring the sensor data for positive blinks.

19. The method of claim 18, wherein monitoring the sensor data to confirm a positive blink further comprises: acquiring a plurality of image frames from a camera facing an eye corresponding to the eye state, and The plurality of image frames are monitored for blinks of the eye.

20. The method of claim 16, further comprising ceasing rendering of the first series of frames upon presentation of the second series of frames.

21. The method of claim 16, wherein obtaining additional details for the virtual content comprises detecting completion of downloading of the additional details.

22. A non-transitory computer readable medium comprising computer readable code for executing the method according to any one of claims 15 to 21.

23. A system for triggering a state change, comprising: one or more processors; and One or more computer-readable media comprising computer-readable code executable by the one or more processors to: rendering a first series of frames for presentation to a user, the first series of frames comprising virtual content at a first level of detail; obtaining additional details for the virtual content; monitoring an eye state of the user's eyes to detect an anticipated inattention state based on obtaining the additional details for the virtual content; responsive to detecting the expected inattention state, rendering a second series of frames, the second series of frames including the virtual content with the additional detail, wherein the second series of frames are rendered during presentation of the first series of frames; as well as The determination that the state of the user's eyes is indicative of an inattentive state of the user causes the presentation of the first series of frames to be replaced with the second series of frames.

24. The system of claim 23, wherein the computer readable code for monitoring an eye state comprises computer readable code for: capturing sensor data corresponding to the user's eyes; and Based on the sensor data, it is determined that the user is about to enter the inattentive state.

25. The system of claim 24, wherein the computer readable code for detecting an inattentive state comprises computer readable code for monitoring a gaze direction of the eye to confirm that the eye is not gazing at a display device.

26. The system of claim 24, wherein the computer readable code that detects an inattentive state comprises computer readable code that monitors the sensor data for positive eye blinks.

27. The system of claim 23, further comprising computer readable code for ceasing rendering of the first series of frames in response to presentation of the second series of frames.

28. The system of claim 23, wherein the computer readable code for obtaining additional details for the virtual content comprises computer readable code for detecting completion of downloading of the additional details.

Images