Gaze activation on the display interface

By combining gaze vectors and head posture information, the HUD interface is activated using a combination of gaze direction and head movement, solving the problem of unintentional activation, improving user experience, and extending battery life.

CN117396831BActive Publication Date: 2026-11-17APPLE INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202280038286.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-05-28
Filing Date
2022-05-13
Publication Date
2026-11-17
Estimated Expiration
2042-05-13

AI Technical Summary

Technical Problem

In the prior art, gaze tracking devices are prone to unintentional activation and false positive input when activating the head-up display (HUD) interface, resulting in a degraded user experience and shortened battery life.

Method used

By combining gaze vectors and head posture information, the HUD interface is activated using a combination of gaze direction and specific head movements, reducing unintentional activation, such as gazing in a specific direction and performing head movements like nodding.

Benefits of technology

It effectively reduces unintentional activation of the HUD interface, reduces the amount of user input, improves user experience, and extends battery life.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117396831B_ABST
    Figure CN117396831B_ABST
Patent Text Reader

Abstract

The various embodiments disclosed herein include devices, systems, and methods for activating a display interface in an environment using gaze vectors and head pose information. In some embodiments, the device includes a sensor for sensing a user's head pose, a display, one or more processors, and a memory. In various embodiments, the method includes displaying an environment comprising a field of view. Based on the gaze vector, a first position within the field of view is determined where the user's gaze is directed. A head pose value corresponding to the user's head pose is obtained. A user interface is displayed in the environment given that the head pose value corresponds to a movement of the user's head toward the first position.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 194,528, filed May 28, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates in general to interacting with computer-generated content. Background Technology

[0004] Some devices are capable of generating and rendering graphical environments that include numerous objects. These objects can mimic real-world objects. These environments can be rendered on mobile communication devices. Attached Figure Description

[0005] Therefore, this disclosure will be understood by those skilled in the art, and a more detailed description can be made with reference to some exemplary embodiments, some of which are shown in the accompanying drawings.

[0006] Figures 1A to 1G It is a diagram based on an exemplary operating environment of some specific implementations.

[0007] Figure 2 It is a block diagram based on some specific implementations of the display interface engine.

[0008] Figures 3A to 3C This is a flowchart representation of a method for activating a head-up display (HUD) interface in an extended reality (XR) environment using gaze vectors and head pose information, based on some specific implementations.

[0009] Figure 4 This is a block diagram of a device that activates the HUD interface in an XR environment based on specific implementations using gaze vectors and head pose information.

[0010] Figures 5A to 5H It is a diagram based on an exemplary operating environment of some specific implementations.

[0011] Figure 6 It is a block diagram based on some specific implementations of the display interface engine.

[0012] Figures 7A to 7B It is a flowchart representation of a method for activating the HUD interface using the first and second user focus positions according to some specific implementations.

[0013] Figure 8 It is a block diagram of a device that uses the first and second user focus positions to activate the HUD interface according to some specific implementations.

[0014] Figure 9It is a flowchart representation of a method for displaying a user interface based on gaze and head movements, according to some specific implementations.

[0015] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Additionally, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Summary of the Invention

[0016] The various embodiments disclosed herein include devices, systems, and methods for activating a head-up display (HUD) interface in an extended reality (XR) environment using gaze vectors and head pose information. In some embodiments, the device includes sensors for sensing a user's head pose, a display, one or more processors, and memory. In various embodiments, the method includes displaying an XR environment comprising a field of view. Based on the gaze vector, a first position within the field of view is determined where the user's gaze is directed. A head pose value corresponding to the user's head pose is obtained. A user interface is displayed in the XR environment, provided that the head pose value corresponds to a movement of the user's head toward the first position.

[0017] In various specific implementations, a method includes obtaining a first user input corresponding to a first user focus position. Determining that the first user focus position corresponds to a first position within a field of view. Obtaining a second user input corresponding to a second user focus position. Displaying a first user interface provided that the second user focus position corresponds to a second position within the field of view that is different from the first position.

[0018] According to some embodiments, the device includes one or more processors, non-transitory memory, and one or more programs. In some embodiments, the one or more programs are stored in the non-transitory memory and executed by one or more processors. In some embodiments, the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause to perform any of the methods described herein. According to some embodiments, a device includes one or more processors, non-transitory memory, and means for performing or causing to perform any of the methods described herein. Detailed Implementation

[0019] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0020] People can sense or interact with the physical environment or world without using electronic devices. Physical features such as physical objects or surfaces can be included within the physical environment. For example, a physical environment may correspond to a physical city with physical buildings, roads, and vehicles. People can directly perceive or interact with the physical environment through various means such as smell, sight, taste, hearing, and touch. This is in contrast to extended reality (XR) environments, which can refer to partially or fully simulated environments that people can sense or interact with using electronic devices. XR environments can include virtual reality (VR) content, mixed reality (MR) content, augmented reality (AR) content, and so on. Using an XR system, physical movement of a person or a portion of their representation can be tracked, and in response, the properties of virtual objects in the XR environment can be altered in a manner that conforms to at least one law of nature. For example, an XR system can detect head movement of a user and adjust the auditory and graphical content presented to the user in a way that simulates how sound and view would change in a physical environment. In other examples, an XR system can detect movement of electronic devices (e.g., laptops, tablets, mobile phones, etc.) that present the XR environment. Therefore, XR systems can simulate how sound and view will change in the physical environment to adjust the auditory and graphical content presented to the user. In some instances, other inputs, such as representations of body movement (e.g., voice commands), can enable XR systems to adjust the properties of the graphical content.

[0021] Numerous types of electronic systems allow users to sense or interact with their XR environment. An incomplete list of examples includes lenses with integrated display capabilities placed on the user's eyes (e.g., contact lenses), head-up displays (HUDs), projection-based systems, head-mounted systems, windows or windshields with integrated display technology, headsets / earpieces, input systems with or without haptic feedback (e.g., handheld or wearable controllers), smartphones, tablets, desktop / laptop computers, and speaker arrays. Head-mounted systems may include opaque displays and one or more speakers. Other head-mounted systems may be configured to receive from opaque external displays, such as the opaque external display of a smartphone. Head-mounted systems may use one or more image sensors to capture images / video of the physical environment, or one or more microphones to capture audio of the physical environment. Some head-mounted systems may include transparent or translucent displays instead of opaque displays. Transparent or translucent displays may guide light representing an image to the user's eyes through media such as holographic media, optical waveguides, optical combiners, optical reflectors, other similar technologies, or combinations thereof. Various display technologies can be used, such as liquid crystal on silicon, LED, uLED, OLED, laser scanning light sources, digital light projection, or combinations thereof. In some examples, transparent or translucent displays can be selectively controlled to become opaque. Projection-based systems can utilize retinal projection technology, which projects images onto a user's retina, or can project virtual content into a physical environment, such as onto a physical surface or as a hologram.

[0022] The specific implementations described herein envision using gaze information to determine the virtual object on which a user's attention is focused. Implementers should consider the extent to which gaze information is collected, analyzed, disclosed, transmitted, and / or stored in order to respect established privacy policies and / or privacy practices. These considerations should include practices that the application generally considers to comply with or exceed industry and / or government requirements for protecting user privacy. This disclosure also envisions that the use of user gaze information may be limited to the extent necessary to achieve the described implementations. For example, in an implementation where the user's device provides processing power, gaze information may be processed locally on the user's device.

[0023] Some devices display extended reality (XR) environments that include one or more objects (e.g., virtual objects). Users can select objects or otherwise interact with them through various modalities. For example, some devices allow users to select objects or otherwise interact with them using gaze input. Gaze-tracking devices (such as user-facing image sensors) acquire images of the user's pupils. This image can be used to determine gaze vectors. Gaze-tracking devices can use gaze vectors to determine which object the user intends to select or interact with.

[0024] When using gaze-tracking devices, users may find it beneficial to have convenient access to certain user interface elements, such as widgets, information elements, and / or shortcuts to frequently accessed applications. Gaze-tracking devices may present interfaces, including but not limited to heads-up display (HUD) interfaces that incorporate one or more user interface elements. Users may unintentionally trigger activation of the HUD interface. For example, gaze-tracking devices may register false positive inputs, such as user activation of the HUD interface when the user does not actually intend to activate it. When this occurs, the user may expend effort (e.g., additional user input) to dismiss the HUD interface. Additionally, users may unintentionally interact with elements of the HUD interface. For example, a user may unintentionally activate controls that form part of the HUD interface. These unintentional interactions can degrade the user experience. Power consumption may be adversely affected by additional inputs involved in correcting false positives.

[0025] This disclosure provides methods, systems, and / or devices for activating a head-up display (HUD) interface in an XR environment using a combination of gaze vectors and head pose information. In some embodiments, the device displays the HUD interface when a user gazes in a specific direction (e.g., upwards or to the upper left corner of the field of view) and performs a head movement in the same direction as the gaze. The device can train the user to perform this combination of gaze and head movement by displaying an indicator (e.g., a red dot) that prompts the user to look at the indicator and then instructing the user to perform a head movement (e.g., nodding). In some embodiments, the device may omit displaying the indicator. For example, the indicator may be gradually phased out and eventually omitted as the user becomes more familiar with the technology.

[0026] In some implementations, using a combination of gaze vectors and head pose information to activate the HUD interface improves the user experience, for example, by reducing unintentional activation of the HUD interface. The amount of user-provided input can be reduced, for example, by reducing the amount of input required to correct false positives. Therefore, battery life can be enhanced.

[0027] Figure 1AThis is a block diagram of an exemplary operating environment 10 according to some embodiments. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for brevity and to avoid obscuring further relevant aspects of the exemplary embodiments disclosed herein. Therefore, as a non-limiting example, the operating environment 10 includes an electronic device 100 and a display interface engine 200. In some embodiments, the electronic device 100 includes a handheld computing device that can be held by a user 20. For example, in some embodiments, the electronic device 100 includes a smartphone, tablet computer, media player, laptop computer, etc. In some embodiments, the electronic device 100 includes a wearable computing device that can be worn by the user 20. For example, in some embodiments, the electronic device 100 includes a head-mounted device (HMD) or electronic watch.

[0028] exist Figure 1A In the example, the display interface engine 200 resides at the electronic device 100. For example, the electronic device 100 implements the display interface engine 200. In some implementations, the electronic device 100 includes a set of computer-readable instructions corresponding to the display interface engine 200. Although the display interface engine 200 is shown as being integrated into the electronic device 100, in some implementations, the display interface engine 200 is separate from the electronic device 100. For example, in some implementations, the display interface engine 200 resides at another device (e.g., at a controller, server, or cloud computing platform).

[0029] like Figure 1A As shown, in some embodiments, electronic device 100 presents an extended reality (XR) environment 106 that includes the field of view of user 20. In some embodiments, the XR environment 106 is referred to as a computer graphics environment. In some embodiments, the XR environment 106 is referred to as a graphics environment. In some embodiments, electronic device 100 generates the XR environment 106. Alternatively, in some embodiments, electronic device 100 receives the XR environment 106 from another device that generates the XR environment 106.

[0030] In some embodiments, XR environment 106 includes a virtual environment that is a simulated replacement of the physical environment. In some embodiments, XR environment 106 is synthesized by electronic device 100. In such embodiments, XR environment 106 is different from the physical environment in which electronic device 100 resides. In some embodiments, XR environment 106 includes an enhanced environment that is a modified version of the physical environment. For example, in some embodiments, electronic device 100 modifies (e.g., enhances) the physical environment in which electronic device 100 resides to generate XR environment 106. In some embodiments, electronic device 100 generates XR environment 106 by simulating a copy of the physical environment in which electronic device 100 resides. In some embodiments, electronic device 100 generates XR environment 106 by removing and / or adding items from the simulated copy of the physical environment in which electronic device 100 resides.

[0031] In some implementations, the XR environment 106 includes various virtual objects, such as XR object 110 (“object 110”, hereinafter referred to as “object 110”). In some implementations, the XR environment 106 includes multiple objects. Figure 1A In the example, XR environment 106 includes objects 110, 112, and 114. In some embodiments, virtual objects are referred to as graphical objects or XR objects. In various embodiments, electronic device 100 obtains virtual objects from object data storage (not shown). For example, in some embodiments, electronic device 100 retrieves object 110 from object data storage. In some embodiments, virtual objects represent physical items. For example, in some embodiments, virtual objects represent equipment (e.g., machines, such as airplanes, tanks, robots, motorcycles, etc.). In some embodiments, virtual objects represent fictional elements (e.g., entities derived from fictional materials, such as action figures or fictional equipment such as flying motorcycles).

[0032] In various embodiments, as illustrated in Figure 1B, electronic device 100 (e.g., display interface engine 200) determines gaze vector 120. For example, electronic device 100 may include a user-facing image sensor (e.g., a forward-facing or inward-facing camera). In some embodiments, the user-facing image sensor captures one or more images of the user 20's eyes. Electronic device 100 may determine gaze vector 120 based on one or more images. Based on gaze vector 120, electronic device 100 may determine that the user 20's gaze is directed at a specific location 122 within the XR environment 106. In some embodiments, electronic device 100 may display a visual effect 124 in conjunction with location 122. For example, electronic device 100 may display an area of ​​increased brightness around location 122. As another example, electronic device 100 may display a pointer at or near location 122.

[0033] In some specific implementations, such as Figure 1C As shown, electronic device 100 (e.g., display interface engine 200) obtains head pose value 130 corresponding to head pose 132 of user 20. For example, electronic device 100 may include one or more sensors configured to sense the position and / or movement of user 20's head. The one or more sensors may include, for example, an image sensor, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU).

[0034] In some implementations, as shown in Figure ID, the electronic device 100 displays the user interface 140 in the XR environment 106 when the head pose value 130 corresponds to the movement of the user 20's head toward position 122. For example, if position 122 is in the upper right corner of the user 20's field of view, the electronic device 100 may display the user interface 140 if the head pose value 130 corresponds to the movement of the user 20's head in directions including tilting upwards and rotating or translating to the right. In some implementations, the electronic device 100 displays the user interface 140 in response to a head pose value 130 corresponding to a predefined head movement (e.g., nodding) when the gaze vector 120 indicates that the user 20's gaze is directed toward position 122. Thus, in some implementations, the user 20 can trigger the display of the user interface 140 by simultaneously gazing at position 122 and performing a nod. In some implementations, if the user 20 gazes at position 122 and does not perform a nod, the electronic device 100 does not display the user interface 140.

[0035] In some embodiments, the user interface 140 is displayed near location 122 and includes one or more user interface elements. For example, the user interface 140 may include an information element 142 that displays information, for example, from an application running on electronic device 100 and / or another device. In some embodiments, the user interface 140 includes a power indicator 144. The user 20 may provide input to the power indicator 144 and control the application running on electronic device 100 and / or another device. In some embodiments, the user interface 140 includes a shortcut 146. The user 20 may provide input to the shortcut 146 and open the application running on electronic device 100 and / or another device, and / or access content items stored on electronic device 100 and / or another device.

[0036] In some embodiments, as shown in Figure IE, electronic device 100 displays an indication 150 at position 122. The indication 150 can be used to train user 20 to perform head movements to generate head pose values ​​that cause user interface 140 to be displayed in XR environment 106. In some embodiments, electronic device 100 displays indication 150 in XR environment 106. The indication 150 can be obscured (e.g., stopped from display) after a condition is met. For example, electronic device 100 may obscure indication 150 after user interface 140 has been displayed in XR environment 106 for a duration of a threshold. In some embodiments, electronic device 100 obscures indication 150 after user interface 140 has been displayed in XR environment 106 at least a threshold number of times. In some embodiments, electronic device 100 obscures (e.g., stops displaying) indication 150 in response to detecting user input for indication 150. For example, when user 20 makes a gesture or head movement toward display 150, display 150 may be obscured (e.g., stopped from display) in XR environment 106. In some embodiments, electronic device 100 abandons display of display 150 in response to determining that the user interface activation score is greater than a threshold activation score (e.g., the activation rate of user interface 140 is greater than the threshold activation rate), indicating that the user has become accustomed to activating the display of user interface 140 using a combination of gaze input and head movement.

[0037] In some embodiments, as shown in Figure IF, electronic device 100 alters one or more visual attributes of user interface 140. For example, electronic device 100 may alter one or more visual attributes of user interface 140 to enhance its visibility. In some embodiments, electronic device 100 displays visual effects 160 in conjunction with user interface 140. For example, electronic device 100 may change the brightness of user interface 140. In some embodiments, electronic device 100 alters the contrast between user interface 140 and XR environment 106 (e.g., a transparent portion of XR environment 106). In some embodiments, electronic device 100 changes the color of user interface 140, for example, to enhance its visibility. In some embodiments, electronic device 100 changes the size of user interface 140. For example, electronic device 100 may display user interface 140 at a larger size. In some embodiments, electronic device 100 displays animations in conjunction with user interface 140.

[0038] In some specific implementations, such as Figure 1GThis indicates that the electronic device 100 may, for example, obscure the user interface 140 after a disengagement condition has occurred. For example, the electronic device 100 may remove (e.g., stop displaying) the user interface 140 from the XR environment 106 after a threshold duration has elapsed following the detection of user input to the user interface 140. In some implementations, the electronic device 100 may remove the user interface 140 from the XR environment 106 after the user interface 140 has been displayed for a threshold duration. In some implementations, the electronic device 100 may remove the user interface 140 from the XR environment 106 in response to the detection of, for example, specific user input to the user interface 140. For example, the user may perform a specified gesture (e.g., limb or head movement) causing the user interface 140 to disengage.

[0039] In some implementations, the user interface 140 is obscured by removing it from the XR environment 106. This obscuration can be achieved by altering one or more visual properties of the user interface 140 to make it less prominent in the XR environment 106. For example, the electronic device 100 may reduce the brightness of the user interface 140. As another example, the electronic device 100 may increase the transparency of the user interface 140. In some implementations, the electronic device 100 reduces the contrast between the user interface 140 and the XR environment 106 (e.g., a transparent portion of the XR environment 106). In some implementations, the electronic device 100 changes the color of the user interface 140, for example, to reduce its visibility. In some implementations, the electronic device 100 changes the size of the user interface 140. For example, the electronic device 100 may display the user interface 140 in a smaller size.

[0040] In some embodiments, electronic device 100 includes or is attached to a head-mounted device (HMD) worn by user 20. According to various embodiments, the HMD presents (e.g., displays) an XR environment 106. In some embodiments, the HMD includes an integrated display (e.g., a built-in display) that displays the XR environment 106. In some embodiments, the HMD includes a head-mountable housing. In various embodiments, the head-mountable housing includes an attachment area to which another device having a display can be attached. For example, in some embodiments, electronic device 100 can be attached to the head-mountable housing. In various embodiments, the head-mountable housing is shaped to form a receiver for receiving another device (e.g., electronic device 100) including a display. For example, in some embodiments, electronic device 100 slides / snapes onto or otherwise attaches to the head-mountable housing. In some embodiments, the display of the device attached to the head-mountable housing presents (e.g., displays) the XR environment 106. In various specific implementations, examples of electronic devices 100 include smartphones, tablets, media players, laptops, etc.

[0041] Figure 2 A block diagram of a display interface engine 200 according to some specific embodiments is shown. In some embodiments, the display interface engine 200 includes an environment renderer 210, an image data acquirer 220, a head pose value acquirer 230, and a user interface generator 240. In various embodiments, the environment renderer 210 displays an extended reality (XR) environment including a set of virtual objects in the field of view. For example, see reference... Figure 1A The environment renderer 210 can display the XR environment 106 on the monitor 212, including virtual objects 110, 112, and 114. In various embodiments, the environment renderer 210 obtains the virtual objects from the object data storage 214. The virtual objects may represent physical items. For example, in some embodiments, virtual objects represent equipment (e.g., machines, such as airplanes, tanks, robots, motorcycles, etc.). In some embodiments, virtual objects represent fictional elements.

[0042] In some specific implementations, the image data acquirer 220 acquires data from the captured user (e.g., Figure 1A One or more image sensors 222 acquire sensor data from one or more images of user 20. For example, a user-facing image sensor (e.g., a forward-facing camera or an inward-facing camera) can capture one or more images of user 20's eyes and generate image data 224. Image data acquirer 220 acquires image data 224. In some specific implementations, image data acquirer 220 determines gaze vector 226 based on image data 224. Display interface engine 200 can determine the location of user 20's gaze within the field of view based on gaze vector 226.

[0043] In some implementations, the head pose value acquirer 230 acquires head sensor data 232 from one or more head position sensors 234 that sense the position and / or movement of the user 20's head. The one or more head position sensors 234 may include, for example, an image sensor, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU). The head pose value acquirer 230 may generate a head pose value 236 based on the head sensor data 232.

[0044] In some implementations, the user interface generator 240 causes the user interface to be displayed in the XR environment 106 when the head pose value 236 corresponds to the movement of the user 20's head toward the position pointed to by the user 20's gaze. For example, if the position is in the upper right corner of the user 20's field of view, the user interface generator 240 may generate a user interface and insert it into the XR environment 106, which will be rendered by the environment renderer 210, if the head pose value 236 corresponds to the movement of the user 20's head in directions including tilting upwards and rotating or translating to the right. In some implementations, the user interface generator 240 modifies the XR environment 106 to generate a modified XR environment that includes a representation of the user interface. In some implementations, the user interface generator 240 triggers the display of the user interface in response to the simultaneous detection of a head pose value 236 corresponding to a threshold head movement (e.g., nodding) and a gaze vector 226 indicating that the user's gaze is pointing toward a specific position (e.g., the position associated with the user interface). In some implementations, if gaze vector 226 indicates that the user's gaze is directed at a specific location associated with the user interface, but head pose value 236 does not correspond to a threshold head movement (e.g., the user is gazing at the upper right corner but not nodding), then the user interface generator 240 does not trigger the display of the user interface. Similarly, in some implementations, if gaze vector 226 indicates that the user's gaze is not directed at a specific location associated with the user interface, but head pose value 236 corresponds to a threshold head movement (e.g., the user is not gazing at the upper right corner but nodding), then the user interface generator 240 does not trigger the display of the user interface.

[0045] In some embodiments, the environment renderer 210 and / or user interface generator 240 displays an indication of ability to train a user to perform head movements that produce head pose values ​​that cause the user interface to be displayed in the XR environment 106. For example, the indication of ability may be displayed when the image data acquirer 220 determines that the gaze vector 226 is pointing to a specific location in the XR environment 106. The indication of ability may prompt the user 20 to perform head movements toward the indication of ability. In some embodiments, the indication of ability is masked after a condition is met (e.g., the display of the indication of ability is stopped). For example, the indication of ability may be masked after the user interface has been displayed in the XR environment 106 for a duration of a threshold. In some embodiments, the indication of ability is masked after the user interface has been displayed in the XR environment 106 at least a threshold number of times. In some embodiments, the indication of ability is masked in response to the detection of user input (e.g., posture or head movement) toward the indication of ability.

[0046] In some embodiments, the user interface generator 240 enhances the visibility of the user interface by altering one or more visual attributes of the user interface. For example, the user interface generator 240 may change the brightness of the user interface. In some embodiments, the user interface generator 240 changes the contrast between the user interface and the XR environment 106 (e.g., a transparent portion of the XR environment 106). In some embodiments, the user interface generator 240 changes the color of the user interface. In some embodiments, the user interface generator 240 increases the size of the user interface.

[0047] In some implementations, the user interface generator 240 removes or reduces the visibility of the user interface, for example, after a dissolution condition has occurred. For example, the user interface generator 240 may remove the user interface from the XR environment 106 after a threshold duration has elapsed since user input to the user interface was detected. In some implementations, the user interface generator 240 removes the user interface from the XR environment 106 after the user interface has been displayed for a threshold duration. In some implementations, the user interface generator 240 removes the user interface from the XR environment 106 in response to detecting, for example, specific user input to the user interface. For example, a user may perform a specified gesture (e.g., limb or head movement) that causes the user interface to dissolve.

[0048] The visibility of the user interface can be reduced by altering one or more visual attributes of the user interface. For example, the user interface generator 240 can reduce the brightness of the user interface. As another example, the user interface generator 240 can increase the transparency of the user interface. In some implementations, the user interface generator 240 reduces the contrast between the user interface and the XR environment 106 (e.g., a transparent portion of the XR environment 106). In some implementations, the user interface generator 240 changes the color of the user interface. In some implementations, the user interface generator 240 reduces the size of the user interface.

[0049] Figures 3A to 3C This is a flowchart representation of method 300 for activating a head-up display (HUD) interface in an extended reality (XR) environment using gaze vectors and head pose information. In various specific implementations, method 300 is performed by a device (e.g., Figures 1A to 1G The electronic device 100 shown, or Figures 1A to 1G and Figure 2 The method 300 is executed by the display interface engine 200 shown. In some embodiments, method 300 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 300 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0050] In various implementations, an XR environment including the field of view is shown. In some implementations, the XR environment is generated. In some implementations, the XR environment is received from another device that generated the XR environment.

[0051] An XR environment may include a virtual environment, which is a simulated replacement of the physical environment. In some implementations, the XR environment is composite and differs from the physical environment in which the electronic device resides. In some implementations, the XR environment includes an enhanced environment, which is a modified version of the physical environment. For example, in some implementations, the electronic device modifies the physical environment in which it resides to generate the XR environment. In some implementations, the electronic device generates the XR environment by simulating a copy of the physical environment in which it resides. In some implementations, the electronic device removes and / or adds items from a simulated copy of the physical environment in which it resides to generate the XR environment.

[0052] In some embodiments, the electronic device includes a head-mounted device (HMD). The HMD may include an integrated display (e.g., a built-in display) for displaying an XR environment. In some embodiments, the HMD includes a head-mountable housing. In various embodiments, the head-mountable housing includes an attachment area to which another device having a display can be attached. In various embodiments, the head-mountable housing is shaped to form a receiver for receiving another device including a display. In some embodiments, the display of the device attached to the head-mountable housing presents (e.g., displays) the XR environment. Examples of electronic devices in various embodiments include smartphones, tablets, media players, laptops, etc.

[0053] In various embodiments, as illustrated in box 320, method 300 includes determining a first location within the user's gaze-directed field of view based on a gaze vector. For example, in some embodiments, a user-facing image sensor (such as a forward-facing or inward-facing camera) is used to capture one or more images of the user's eyes. The gaze vector can be determined based on one or more images. In some embodiments, as illustrated in box 320a, method 300 includes determining a second location associated with the gaze vector. For example, an electronic device can determine the location within an XR environment to which the gaze vector is directed.

[0054] In some implementations, the electronic device determines that the gaze vector points to a specific location, such as a corner of the field of view. For example, as shown in box 320b, method 300 may include determining that the user's gaze is directed to the first location if a second location associated with the gaze vector meets a proximity criterion relative to a first location. In some implementations, as shown in box 320c, method 300 may include determining that the user's gaze is directed to the first location if the second location associated with the gaze vector meets a proximity threshold duration. For example, if the gaze vector points to a second location near the first location for a duration less than the threshold duration (e.g., the user only glances toward the first location), the electronic device may abandon determining that the user's gaze is directed to the first location.

[0055] In some embodiments, as indicated by box 320d, the electronic device displays an indication (e.g., a dot) near a first position. The indication may trigger head movement corresponding to a head pose value that causes the user interface to be displayed in an XR environment. For example, the indication may be displayed as a target, such as a dot in the XR environment. In some embodiments, as indicated by box 320f, the method includes stopping the display of the indication after a condition is met. For example, the electronic device may stop displaying the indication after the user interface has been displayed for a duration of a threshold, as indicated by box 320g. In some embodiments, as indicated by box 320h, the electronic device stops displaying the indication after the user interface has been displayed a threshold number of times. In some embodiments, as indicated by box 320i, the electronic device stops displaying the indication in response to detecting user input (such as gesture or head movement) to the indication.

[0056] In various specific implementations, such as Figure 3B As indicated by box 330, method 300 includes obtaining a head pose value corresponding to a user's head pose. In some embodiments, as shown in box 330a, the head pose value corresponds to sensor data associated with a sensor. For example, the electronic device may include one or more sensors configured to sense the position and / or movement of the user's head. In some embodiments, as shown in box 330b, the sensor data includes IMU data obtained from an inertial measurement unit (IMU). As shown in box 330c, in some embodiments, the sensor includes an accelerometer. In some embodiments, as shown in box 330d, the sensor includes a gyroscope. As shown in box 330e, in some embodiments, the sensor includes a magnetometer.

[0057] In various embodiments, as indicated by box 340, method 300 includes displaying a user interface in an XR environment under the condition that the head pose value corresponds to a rotation of the user's head toward a first position. For example, if the first position is the upper right corner of the field of view, the electronic device displays the user interface if the user's gaze is directed toward the upper right corner of the field of view and the user performs a head rotation toward the upper right corner of the field of view. In some embodiments, the condition is a rotation of the head forward vector toward the first position. In some embodiments, the head forward vector indicates the direction the user's head is facing. In some embodiments, using a combination of gaze vector and head pose information to activate the HUD interface improves the user experience, for example, by reducing unintentional activation of the HUD interface. The amount of user input provided by the user can be reduced, for example, by reducing the amount of input required to correct false positives. Therefore, battery life can be enhanced.

[0058] In some embodiments, visual attributes of the user interface are altered to enhance or reduce its visibility. For example, as shown in box 340a, the visual attributes of the user interface can be changed based on a gaze vector. This can be done when the user is looking at the user interface, causing it to be displayed more prominently. In some embodiments, as shown in box 340b, visual attributes include the brightness of the user interface. In some embodiments, as shown in box 340c, visual attributes include the contrast of the user interface, such as a transparent portion of a reference XR environment. In some embodiments, as shown in box 340d, visual attributes include the color of the user interface. For example, the color of the user interface can be changed to enhance its visibility. In some embodiments, as shown in box 340e, visual attributes include the size of the user interface. For example, an electronic device can display the user interface at a larger size.

[0059] In some specific implementations, as indicated in box 340f, the electronic device obscures the user interface. For example, as indicated in box 340g, the electronic device may obscure the user interface after the user interface has been displayed for a duration that meets a threshold threshold. Figure 3C As indicated in box 340h, the electronic device can obscure the user interface after a threshold duration has elapsed following the detection of user input to the user interface. In some specific implementations, as indicated in box 340i, the electronic device obscures the user interface in response to the detection of user input to the user interface. For example, the user may perform a specified gesture (e.g., limb or head movement) to cause the user interface to be disengaged.

[0060] In some implementations, as illustrated in box 340j, the user interface is obscured by stopping its display. For example, the user interface generator 240 can modify the XR environment so that it no longer includes a representation of the user interface. The environment renderer 210 can display the XR environment without a user interface.

[0061] As indicated by box 340k, the user interface can be obscured by altering its visual attributes, for example, to make it less prominent in an XR environment. In some embodiments, as indicated by box 3401, the visual attributes include the brightness of the user interface. For example, the electronic device may reduce the brightness of the user interface. In some embodiments, as indicated by box 340m, the visual attributes include the contrast of the user interface, for example, relative to a transparent portion of the XR environment. In some embodiments, as indicated by box 340n, the visual attributes include the color of the user interface. For example, the electronic device 100 may display the user interface 140 in a smaller size.

[0062] Figure 4 This is a block diagram based on some specific implementations of device 400. In some specific implementations, device 400 implements... Figures 1A to 1G The electronic device 100 shown, and / or Figures 1A to 1G and Figure 2 The display interface engine 200 is shown. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring more relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, device 400 includes one or more processing units (CPUs) 401, a network interface 402, a programming interface 403, a memory 404, one or more input / output (I / O) devices 410, and one or more communication buses 405 for interconnecting these and various other components.

[0063] In some implementations, a network interface 402 is provided to establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices, among other uses. In some implementations, one or more communication buses 405 include circuitry for interconnecting and communicating between system components. Memory 404 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 404 optionally includes one or more storage devices remotely located to one or more CPUs 401. Memory 404 includes a non-transitory computer-readable storage medium.

[0064] In some embodiments, memory 404 or a non-transitory computer-readable storage medium of memory 404 stores programs, modules, and data structures, or subsets thereof, including optional operating system 406, environment renderer 210, image data acquirer 220, head pose value acquirer 230, and user interface generator 240. In various embodiments, device 400 executes... Figures 3A to 3C Method 300 is shown.

[0065] In some implementations, the environment renderer 210 displays an extended reality (XR) environment comprising a set of virtual objects within the field of view. In some implementations, the environment renderer 210 includes instructions 210a, as well as heuristics and metadata 210b.

[0066] In some specific implementations, the image data acquirer 220 acquires data from the captured user (e.g., Figure 1A The image data acquisition unit 220 acquires sensor data from one or more images of the user (user 20). In some embodiments, the image data acquisition unit 220 determines the gaze vector. In some embodiments, the image data acquisition unit 220 performs actions by... Figures 3A to 3C The operation is represented by box 320 in the diagram. For this purpose, the image data acquirer 220 includes instructions 220a, as well as heuristics and metadata 220b.

[0067] In some implementations, the head pose value acquirer 230 acquires head sensor data from one or more head position sensors that sense the position and / or movement of the user 20's head. The one or more head position sensors may include, for example, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU). The head pose value acquirer 230 may generate head pose values ​​based on the head sensor data. In some implementations, the head pose value acquirer 230 performs actions by... Figures 3A to 3C The operation represented by box 330 in the diagram. For this purpose, the head pose value obtainr 230 includes instructions 230a as well as heuristics and metadata 230b.

[0068] In some implementations, the user interface generator 240 causes the user interface to be displayed in the XR environment, provided that the head pose value corresponds to the movement of the user 20's head toward the position pointed to by the user 20's gaze. In some implementations, the user interface generator 240 performs the operation of... Figures 3A to 3C The operation represented by box 340 in the diagram. For this purpose, the user interface generator 240 includes instructions 240a, as well as heuristics and metadata 240b.

[0069] In some implementations, one or more I / O devices 410 include a user-facing image sensor (e.g., Figure 2One or more image sensors 222, which may be implemented as a forward-facing camera or an inward-facing camera. In some specific implementations, one or more I / O devices 410 include one or more head position sensors (e.g., for sensing the position and / or movement of the user's head). Figure 2 One or more head position sensors 234. The one or more head position sensors 234 may include, for example, accelerometers, gyroscopes, magnetometers, and / or inertial measurement units (IMUs). In some embodiments, one or more I / O devices 410 include a display for displaying a graphical environment (e.g., for displaying XR environment 106). In some embodiments, one or more I / O devices 410 include a speaker for outputting auditory signals.

[0070] In various embodiments, one or more I / O devices 410 include a video perspective display that displays at least a portion of the physical environment surrounding device 400 as an image captured by a scene camera. In various embodiments, one or more I / O devices 410 include an optical perspective display that is at least partially transparent and transmits light emitted or reflected by the physical environment.

[0071] It should be understood that Figure 4 This serves as a functional description of various features that may exist in a particular specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 4 Some functional blocks shown individually can be implemented as a single block, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of blocks and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.

[0072] This disclosure provides methods, systems, and / or devices for activating an interface (such as a HUD interface) using a first user focus position and a second user focus position. In some embodiments, the device displays the interface when it receives a first user input associated with a first user focus position corresponding to a first position within the user's field of view, and then receives a second user input associated with a second user focus position corresponding to a second position within the user's field of view. For example, the interface may be activated when the user gazes at a first position (e.g., towards the upper left corner of the field of view) and then gazes at a second position (e.g., towards the upper right corner of the field of view). The device may provide cues to train the user by displaying a capability indicator (e.g., a red dot) at the first position, prompting the user to view the capability indicator. In some embodiments, the device may omit displaying the capability indicator. For example, the capability indicator may be gradually phased out and eventually omitted as the user becomes more familiar with the technology.

[0073] In some implementations, using first and second user focus positions to activate the interface improves the user experience, for example, by reducing unintentional activation of the interface. The amount of user input can be reduced, for example, by reducing the amount of input required to correct false positives. Therefore, battery life can be enhanced.

[0074] Figure 5A This is a block diagram of an example operating environment 500 according to some specific embodiments. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for brevity and to avoid obscuring further relevant aspects of the exemplary embodiments disclosed herein. Therefore, as a non-limiting example, operating environment 500 includes an electronic device 510 and a display interface engine 600. In some embodiments, electronic device 510 includes a handheld computing device that can be held by a user 512. For example, in some embodiments, electronic device 510 includes a smartphone, tablet computer, media player, laptop computer, etc. In some embodiments, electronic device 510 includes a wearable computing device that can be worn by a user 512. For example, in some embodiments, electronic device 510 includes a head-mounted device (HMD) or electronic watch.

[0075] exist Figure 5AIn the example, the display interface engine 600 resides at the electronic device 510. For example, the electronic device 510 implements the display interface engine 600. In some embodiments, the electronic device 510 includes a set of computer-readable instructions corresponding to the display interface engine 600. Although the display interface engine 600 is shown as being integrated into the electronic device 510, in some embodiments, the display interface engine 600 is separate from the electronic device 510. For example, in some embodiments, the display interface engine 600 resides at another device (e.g., at a controller, server, or cloud computing platform).

[0076] like Figure 5A As shown, in some embodiments, electronic device 510 presents an extended reality (XR) environment 514 that includes the field of view of user 512. In some embodiments, the XR environment 514 is referred to as a computer graphics environment. In some embodiments, the XR environment 514 is referred to as a graphics environment. In some embodiments, electronic device 510 generates the XR environment 514. Alternatively, in some embodiments, electronic device 510 receives the XR environment 514 from another device that generated the XR environment 514.

[0077] In some embodiments, the XR environment 514 includes a virtual environment that is a simulated replacement of the physical environment. In some embodiments, the XR environment 514 is synthesized by electronic device 510. In such embodiments, the XR environment 514 is different from the physical environment in which electronic device 510 resides. In some embodiments, the XR environment 514 includes an enhanced environment that is a modified version of the physical environment. For example, in some embodiments, electronic device 510 modifies (e.g., enhances) the physical environment in which electronic device 510 resides to generate the XR environment 514. In some embodiments, electronic device 510 generates the XR environment 514 by simulating a copy of the physical environment in which electronic device 510 resides. In some embodiments, electronic device 510 generates the XR environment 514 by removing and / or adding items from a simulated copy of the physical environment in which electronic device 510 resides.

[0078] In some implementations, the XR environment 514 includes various virtual objects, such as XR object 516 (“object 516”, hereinafter referred to as “object 516”). In some implementations, the XR environment 514 includes multiple objects. Figure 5AIn the example, XR environment 514 includes objects 516, 518, and 520. In some embodiments, the virtual objects are referred to as graphical objects or XR objects. In various embodiments, electronic device 510 obtains the virtual objects from object data storage (not shown). For example, in some embodiments, electronic device 510 retrieves object 516 from object data storage. In some embodiments, the virtual object represents a physical item. For example, in some embodiments, the virtual object represents equipment (e.g., machines, such as airplanes, tanks, robots, motorcycles, etc.). In some embodiments, the virtual object represents a fictional element (e.g., an entity derived from fictional material, such as an action figure or fictional equipment such as a flying motorcycle).

[0079] In various specific implementations, such as Figure 5B The electronic device 510 (e.g., display interface engine 600) receives user input 530 corresponding to user focus position 532. For example, the electronic device 510 may include a user-facing image sensor (e.g., a forward-facing camera or an inward-facing camera). In some embodiments, the user-facing image sensor captures one or more images of the user 512's eyes. The electronic device 510 may determine a gaze vector based on the one or more images. Based on the gaze vector, the electronic device 510 may determine that the user 512's gaze is directed towards user focus position 532. In some embodiments, the electronic device 510 (e.g., display interface 600) obtains head pose values ​​corresponding to the user 512's head posture. For example, the electronic device 510 may include one or more sensors configured to sense the position and / or movement of the user 512's head. The one or more sensors may include, for example, an image sensor, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU). Based on the head pose values, the electronic device 510 may determine that the user 512's head posture is directed towards user focus position 532.

[0080] In some implementations, the electronic device 510 displays a visual effect 534 in conjunction with the user's focus position 532. For example, the electronic device 510 may display an area with increased brightness around the user's focus position 532. As another example, the electronic device 510 may display a pointer at or near the user's focus position 532.

[0081] In some embodiments, electronic device 510 determines a target position 536 in the user's field of view relative to the user 512. This target position 536 may represent a first position for activating the interface. For example, electronic device 510 may determine that the user's focus position 532 corresponds to the target position 536. In some embodiments, electronic device 510 determines that the user's focus position 532 corresponds to the target position 536 if the user's focus position 532 meets a proximity criterion relative to the target position 536. In some embodiments, electronic device 510 determines that the user's focus position 532 corresponds to the target position 536 if the user's focus position 532 meets a proximity criterion duration threshold.

[0082] In some specific implementations, such as Figure 5C The electronic device 510 (e.g., display interface engine 600) receives user input 540 corresponding to user focus position 542. For example, the electronic device 510 may include a user-facing image sensor (e.g., a forward-facing camera or an inward-facing camera). In some embodiments, the user-facing image sensor captures one or more images of the user 512's eyes. The electronic device 510 may determine a gaze vector based on the one or more images. Based on the gaze vector, the electronic device 510 may determine that the user 512's gaze is directed towards the user focus position 542. In some embodiments, the electronic device 510 (e.g., display interface 600) obtains head pose values ​​corresponding to the user 512's head posture. For example, the electronic device 510 may include one or more sensors configured to sense the position and / or movement of the user 512's head. The one or more sensors may include, for example, an image sensor, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU). Based on the head pose values, the electronic device 510 may determine that the user 512's head posture is directed towards the user focus position 542.

[0083] In some implementations, the electronic device 510 displays a visual effect 544 in conjunction with the user's focus position 542. For example, the electronic device 510 may display an area with increased brightness around the user's focus position 542. As another example, the electronic device 510 may display a pointer at or near the user's focus position 542.

[0084] In some implementations, electronic device 510 determines that user input 540 is directed to a target position 546 that is different from target position 536. Target position 546 may represent a second position for confirming activation of the interface. For example, electronic device 510 may determine that user focus position 542 corresponds to target position 546. In some implementations, electronic device 510 determines that user focus position 542 corresponds to target position 546 if user focus position 542 meets a proximity criterion relative to target position 546. In some implementations, electronic device 510 determines that user focus position 542 corresponds to target position 546 if user focus position 542 meets a proximity criterion duration threshold. Although target position 546 is shown as a point on the display of electronic device 510, in other examples, target position 546 may include an area of ​​the display of electronic device 510, a point within XR environment 514, or an area within XR environment 514. In some implementations using head pose, target position 546 may be defined as a direction relative to the initial head pose, such as an upward rotation about the pitch axis.

[0085] In some specific implementations, in response to determining user input 530 Figure 5B The target location 536 shown may be a signal indicator displayed at target location 546. The signal indicator may include any visual indicator, such as a dot, icon, button, etc. Displaying the signal indicator at target location 546 helps the user identify where they should be directed to input to confirm the activation of the interface. In some implementations, the signal indicator may be obscured (e.g., stopped from displaying) in response to a threshold duration or in response to the detection of user input to the signal indicator. For example, the signal indicator may be obscured (e.g., stopped from displaying) when the user 512 makes a gesture toward the signal indicator, stares at the signal indicator, or makes a head movement toward the signal indicator.

[0086] In some implementations, additional information may be presented in response to determining that user input 530 is directed to target location 536 and prior to user input 540 to target location 546. For example, different information or subsets of information available in an interface that the user might want to quickly access, such as time, notifications, messages, etc., may be displayed. In some implementations, the additional information may be masked (e.g., stopped from being displayed) in response to a threshold duration, in response to no user input being detected for the additional information, or in response to detecting user input for an indication located at the aforementioned target location 546.

[0087] In some specific implementations, such as Figure 5DThis means that electronic device 510 displays user interface 550 on a display or in XR environment 514 in response to receiving user input 530 and user input 540 for target location 546. For example, if target location 536 is located near the upper left corner of the field of view and target location 546 is located near the upper right corner of the field of view, then electronic device 510 may display user interface 550 when user 512 gazes at the upper left corner of the field of view and then gazes at the upper right corner of the field of view.

[0088] In some embodiments, the user interface 550 is displayed near the target location 546 and includes one or more user interface elements. For example, the user interface 550 may include an information element 552 that displays information, for example, from an application running on the electronic device 510 and / or another device. In some embodiments, the user interface 550 includes a power indicator 554. The user 512 may provide input to the power indicator 554 and control the application running on the electronic device 510 and / or another device. In some embodiments, the user interface 550 includes a shortcut 556. The user 512 may provide input to the shortcut 556 and open the application running on the electronic device 510 and / or another device, and / or access content items stored on the electronic device 510 and / or another device.

[0089] In some specific implementations, such as Figure 5EThe electronic device 510 displays a power indicator 560 at a target location 536. The power indicator 560 can be used to train the user 512 to provide user input 530 to initiate the process of displaying the user interface 550. In some embodiments, the electronic device 510 displays the power indicator 560 on a display or in an XR environment 514. The power indicator 560 can be masked (e.g., stopped from display) after a condition is met. For example, the electronic device 510 may mask the power indicator 560 after the user interface 550 has been displayed for a threshold duration. In some embodiments, the electronic device 510 may mask the power indicator 560 after the user interface 550 has been displayed a threshold number of times. In some embodiments, the electronic device 510 masks (e.g., stops displaying) the power indicator 560 in response to detecting user input for the power indicator 560. For example, the power indicator 560 may be masked (e.g., stopped from display) when the user 512 makes a gesture toward the power indicator 560, gazes at the power indicator 560, or makes a head movement toward the power indicator 560. In some embodiments, electronic device 510 abandons the display of indication 560 in response to determining that the user interface activation score is greater than a threshold activation score (e.g., the activation rate of user interface 550 is greater than a threshold activation rate), indicating that the user has become accustomed to activating the display of user interface 550 using first and second user focus positions. In some embodiments, indication 560 may be displayed near target location 536 using lights or displays separate from the main display of electronic device 510. For example, LEDs may be positioned around the display and one or more of these LEDs may be illuminated to indicate the direction in which the user should gaze or move their head to initiate interface activation.

[0090] In some specific implementations, such as Figure 5F This indicates that electronic device 510 displays user interface 570 at target location 536. User interface 570 can be used to train user 512 to provide user input 530 to initiate the process of displaying user interface 550. In some embodiments, electronic device 510 displays user interface 570 in XR environment 514. User interface 570 may be visually simpler than user interface 550 (e.g., it may include fewer user interface elements). For example, user interface 570 may include a single user interface element 572, such as an information element, a display, or a shortcut. In some embodiments, user interface 570 is displayed whenever XR environment 514 is displayed. In some embodiments, user interface 570 is displayed when user focus position 532 corresponds to target location 536 (e.g., meets a proximity criterion relative to it). In some embodiments, user interface 570 is displayed when user focus position 532 meets a proximity criterion relative to target location 536 for a duration of a threshold value.

[0091] In some specific implementations, such as Figure 5G This indicates that the electronic device 510 alters one or more visual attributes of the user interface 550, for example, in response to user input to the user interface 550. For example, the electronic device 510 may alter one or more visual attributes of the user interface 550 to enhance its visibility. In some embodiments, the electronic device 510 displays visual effects 580 in conjunction with the user interface 550. For example, the electronic device 510 may change the brightness of the user interface 550. In some embodiments, the electronic device 510 changes the contrast between the user interface 550 and the XR environment 514 (e.g., a transparent portion of the XR environment 514). In some embodiments, the electronic device 510 changes the color of the user interface 550, for example, to enhance its visibility. In some embodiments, the electronic device 510 changes the size of the user interface 550. For example, the electronic device 510 may display the user interface 550 at a larger size. In some embodiments, the electronic device 510 displays animations in conjunction with the user interface 550.

[0092] In some specific implementations, such as Figure 5H This indicates that electronic device 510 may, for example, obscure user interface 550 after a disengagement condition has occurred. For instance, after a threshold duration has elapsed following the detection of user input to user interface 550, electronic device 510 may remove (e.g., stop displaying) user interface 550 from XR environment 514. In some embodiments, electronic device 510 stops displaying user interface 550 after user interface 550 has been displayed for a threshold duration. In some embodiments, electronic device 510 stops displaying user interface 550 in response to the detection of, for example, specific user input to user interface 550. For example, a user may perform a specified gesture (e.g., limb or head movement) causing user interface 550 to be disengaged.

[0093] In some implementations, the user interface 550 is masked by removing it from the XR environment 514 (e.g., by stopping the display of the user interface 550). The user interface 550 can be masked by altering one or more visual properties of the user interface 550 to make it less prominent. For example, the electronic device 510 may reduce the brightness of the user interface 550. As another example, the electronic device 510 may increase the transparency of the user interface 550. In some implementations, the electronic device 510 reduces the contrast between the user interface 550 and the XR environment 514 (e.g., a transparent portion of the XR environment 514). In some implementations, the electronic device 510 changes the color of the user interface 550, for example, to reduce its visibility. In some implementations, the electronic device 510 changes the size of the user interface 550. For example, the electronic device 510 may display the user interface 550 in a smaller size.

[0094] In some embodiments, electronic device 510 includes or is attached to a head-mounted device (HMD) worn by user 512. According to various embodiments, the HMD presents (e.g., displays) an XR environment 514. In some embodiments, the HMD includes an integrated display (e.g., a built-in display) that displays the XR environment 514. In some embodiments, the HMD includes a head-mountable housing. In various embodiments, the head-mountable housing includes an attachment area to which another device having a display can be attached. For example, in some embodiments, electronic device 510 can be attached to the head-mountable housing. In various embodiments, the head-mountable housing is shaped to form a receiver for receiving another device (e.g., electronic device 510) including a display. For example, in some embodiments, electronic device 510 slides / snapes into or otherwise attaches to the head-mountable housing. In some embodiments, the display of the device attached to the head-mountable housing presents (e.g., displays) the XR environment 514. In various specific implementations, examples of electronic devices 510 include smartphones, tablets, media players, laptops, etc.

[0095] Figure 6 This is a block diagram of a display interface engine 600 according to some specific embodiments. In some embodiments, the display interface engine 600 includes an environment renderer 610, an image data acquirer 620, a head pose value acquirer 630, and / or a user interface generator 640. In various embodiments, the environment renderer 610 outputs image data for rendering an extended reality (XR) environment including a set of virtual objects in the field of view. For example, see reference... Figure 5AThe environment renderer 610 can output image data for rendering an XR environment 514 on a display 612, including virtual objects 516, 518, and 520. In various embodiments, the environment renderer 610 obtains virtual objects from an object data memory 614. Virtual objects can represent physical items. For example, in some embodiments, virtual objects represent devices (e.g., machines, such as airplanes, tanks, robots, motorcycles, etc.). In some embodiments, virtual objects represent fictional elements. In some embodiments where the display 612 includes an opaque display, the environment renderer 610 can output image data for representing the XR environment 514 of both virtual and physical objects (e.g., transparent images from an image sensor). In other embodiments where the display 612 includes a transparent or semi-transparent display, the environment renderer 610 can output image data for representing the XR environment 514 of only virtual objects.

[0096] In some specific implementations, the image data acquirer 620 acquires data from the captured user (e.g., Figure 5A One or more image sensors 622 acquire sensor data from one or more images of user 512. For example, a user-facing image sensor (e.g., a forward-facing camera or an inward-facing camera) can capture one or more images of user 512's eyes and can generate image data 624. Image data acquirer 620 can acquire image data 624. In some specific embodiments, image data acquirer 620 determines gaze vector 626 based on image data 624. Display interface engine 600 can determine the location of user 512's gaze direction within the field of view based on gaze vector 626, for example, user focus position 532 and / or user focus position 542.

[0097] In some implementations, the head pose value acquirer 630 acquires head sensor data 632 from one or more head position sensors 634 that sense the position and / or movement of the user 512's head. The one or more head position sensors 634 may include, for example, an image sensor, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU). The head pose value acquirer 630 may generate a head pose value 636 based on the head sensor data 632. The head pose value acquirer 630 may determine the orientation of the head pose value 636 corresponding to the user 512's head facing within the field of view (e.g., user focus position 532 and / or user focus position 542).

[0098] It will be understood that in some embodiments, the image data acquirer 620 or the head pose value acquirer 630 may be omitted. For example, in some embodiments, user input 530 and user input 540 may be gaze input. In such embodiments, the head pose value acquirer 630 may be omitted. As another example, in some embodiments, user input 530 and user input 540 may be head sensor data. In such embodiments, the image data acquirer 620 may be omitted.

[0099] In some implementations, the user interface generator 640 causes, for example, the user interface to be displayed in the XR environment 514 when both the user focus position 532 and the target position 546 are located. For example, if the target position 536 is in the upper left corner of the user 512's field of view and the target position 546 is in the upper right corner of the field of view, then if the user 512 looks towards the upper left corner of the field of view and then towards the upper right corner, the user interface generator 640 can generate the user interface and insert it into the XR environment 514 that will be rendered by the environment renderer 610.

[0100] In some implementations, the user interface generator 640 modifies the XR environment 514 to generate a modified XR environment including a representation of the user interface. In some implementations, the user interface generator 640 triggers the display of the user interface in response to determining that user focus position 532 corresponds to target position 536 and user focus position 542 corresponds to target position 546. In some implementations, if user focus position 532 corresponds to target position 536 but user focus position 542 does not correspond to target position 546 (e.g., user 512 looks at target position 536 but then looks away from target position 546), the user interface generator 640 does not trigger the display of the user interface.

[0101] In some embodiments, the environment renderer 610 and / or user interface generator 640 display an indication of activity, which is used to train user 512 to provide user input 530 to initiate the display of the user interface. The indication of activity may be masked (e.g., stopped from display) after a condition is met. For example, the indication of activity may be masked after the user interface has been displayed for a threshold duration. In some embodiments, the indication of activity may be masked after the user interface has been displayed a threshold number of times. In some embodiments, the indication of activity may be masked in response to the detection of user input to the indication of activity. For example, the indication of activity may be masked when user 512 makes a gesture toward the indication of activity or makes a head movement toward the indication of activity. In some embodiments, the indication of activity may be masked in response to determining that the user interface activation score is greater than a threshold activation score (e.g., the user interface activation rate is greater than a threshold activation rate), indicating that the user has become accustomed to activating the display of the user interface using first and second user focus positions.

[0102] In some embodiments, the user interface generator 640 enhances the visibility of the user interface by altering one or more visual attributes of the user interface. For example, the user interface generator 640 may change the brightness of the user interface. In some embodiments, the user interface generator 640 changes the contrast between the user interface and the XR environment 514 (e.g., a transparent portion of the XR environment 514). In some embodiments, the user interface generator 640 changes the color of the user interface. In some embodiments, the user interface generator 640 increases the size of the user interface.

[0103] In some implementations, the user interface generator 640 removes or reduces the visibility of the user interface, for example, after a dissolution condition has occurred. For example, the user interface generator 640 may stop displaying the user interface after a threshold duration has elapsed since user input to the user interface was detected. In some implementations, the user interface generator 640 stops displaying the user interface after a threshold duration has elapsed since the user interface was displayed. In some implementations, the user interface generator 640 stops displaying the user interface in response to detecting, for example, specific user input to the user interface. For example, a user may perform a specified gesture (e.g., limb or head movement) that causes the user interface to dissolve.

[0104] The visibility of the user interface can be reduced by altering one or more visual attributes of the user interface. For example, the user interface generator 640 can reduce the brightness of the user interface. As another example, the user interface generator 640 can increase the transparency of the user interface. In some implementations, the user interface generator 640 reduces the contrast between the user interface and the XR environment 514 (e.g., a transparent portion of the XR environment 514). In some implementations, the user interface generator 640 changes the color of the user interface. In some implementations, the user interface generator 640 reduces the size of the user interface.

[0105] Figures 7A to 7B This is a flowchart representation of a method 700 for activating an interface using first and second user focus positions, based on some specific implementations. In various specific implementations, method 700 is performed by a device (e.g., Figures 5A to 5H The electronic device 510 shown, or Figures 5A to 5H and Figure 6 The method 700 is executed by the display interface engine 600 shown. In some embodiments, method 700 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, method 700 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0106] In various embodiments, an XR environment including a field of view is shown. In some embodiments, an XR environment is generated. In some embodiments, the XR environment is received from another device that generates the XR environment. The XR environment may include a virtual environment that is a simulated replacement of the physical environment. In some embodiments, the XR environment is synthesized and is different from the physical environment in which the electronic device resides. In some embodiments, the XR environment includes an enhanced environment that is a modified version of the physical environment. For example, in some embodiments, the electronic device modifies the physical environment in which the electronic device resides to generate the XR environment. In some embodiments, the electronic device generates the XR environment by simulating a copy of the physical environment in which the electronic device resides. In some embodiments, the electronic device removes and / or adds items from a simulated copy of the physical environment in which the electronic device resides to generate the XR environment.

[0107] In some embodiments, the electronic device includes a head-mounted device (HMD). The HMD may include an integrated display (e.g., a built-in display) for displaying an XR environment. In some embodiments, the HMD includes a head-mountable housing. In various embodiments, the head-mountable housing includes an attachment area to which another device having a display can be attached. In various embodiments, the head-mountable housing is shaped to form a receiver for receiving another device including a display. In some embodiments, the display of the device attached to the head-mountable housing presents (e.g., displays) the XR environment. Examples of electronic devices in various embodiments include smartphones, tablets, media players, laptops, etc.

[0108] In various embodiments, as indicated by box 710, method 700 includes obtaining a first user input corresponding to a first user focus position. For example, as indicated by box 710a, the first user input may include gaze input. In some embodiments, sensor data may be obtained from one or more image sensors that capture one or more images of the user. For example, a user-facing image sensor (e.g., a forward-facing camera or an inward-facing camera) may capture one or more images of the user's eyes and may generate image data from which a gaze vector can be determined. The gaze vector may correspond to a first user focus position.

[0109] In some implementations, as illustrated in block 710b, the first user input includes head pose input. For example, head sensor data may be obtained from one or more head position sensors that sense the position and / or movement of the user's head. The one or more head position sensors may include, for example, an image sensor, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU). A head pose value may be determined based on the head sensor data. The head pose value may correspond to the orientation of the user's head toward a position within the field of view (e.g., a first user focal position).

[0110] In various embodiments, as indicated by box 720, method 700 includes determining that a first user focus position corresponds to a first position within the field of view. For example, as indicated by box 720a, method 700 may include determining that the first user focus position corresponds to the first position if the first user focus position satisfies a proximity criterion relative to the first position. In some embodiments, as indicated by box 720b, electronic device 510 determines that the first user focus position corresponds to the first position if the first user focus position satisfies a proximity criterion for a duration of a threshold value.

[0111] In some specific implementations, as indicated by box 720c, a second user interface is displayed in response to determining that the first user focus position corresponds to a first position. The second user interface can be used to train the user to initiate the display of the first user interface (e.g., Figure 5DThe second user interface (550) is the process of user input. The second user interface may be visually simpler than the first user interface (e.g., it may include fewer user interface elements). For example, the second user interface may include a single user interface element, such as an information element, a display, or a shortcut. In some embodiments, the second user interface is displayed whenever the XR environment is displayed. In some embodiments, as indicated by box 720d, the second user interface is displayed when the first user focus position meets a proximity criterion duration relative to the first position.

[0112] In some embodiments, as indicated by box 720e, method 700 includes displaying a power indicator near a first location. The power indicator may be displayed before first user input corresponding to a first user focus location is received. The power indicator can be used to train a user to provide user input that initiates the process of displaying a first user interface. As indicated by box 720f, the display of the power indicator may be stopped after a condition is met. For example, as indicated by box 720g, electronic device 510 may stop displaying the power indicator after the first user interface 550 has been displayed for a duration of a threshold duration. In some embodiments, as indicated by box 720h, electronic device 510 stops displaying the power indicator after the first user interface has been displayed a threshold number of times. In some embodiments, as indicated by box 720i, electronic device 510 stops displaying the power indicator in response to detecting first user input for the power indicator. For example, the power indicator may be obscured (e.g., stopped from display) when a user makes a gesture toward the power indicator or makes a head movement toward the power indicator. In some implementations, the electronic device 510 abandons the display of a power indication in response to determining that the user interface activation score is greater than a threshold activation score (e.g., the user interface activation rate is greater than a threshold activation rate), which indicates that the user has become accustomed to using the first and second user focus positions to activate the display of the user interface.

[0113] In various specific implementations, such as Figure 7B As indicated by box 730, method 700 includes obtaining a second user input corresponding to a second user focus position. For example, as indicated by box 730a, the second user input may include gaze input. In some specific embodiments, sensor data may be obtained from one or more image sensors that capture one or more images of the user. For example, a user-facing image sensor (e.g., a forward-facing camera or an inward-facing camera) may capture one or more images of the user's eyes and may generate image data from which a gaze vector can be determined. The gaze vector may correspond to a second user focus position.

[0114] In various embodiments, as indicated by box 730b, a power indicator is displayed at a second user focus position, for example, before receiving second user input. For example, a point may be displayed at the second user focus position in response to a first user focus position corresponding to a first position. The power indicator may provide a visual cue to the user to inform the user of the second position to be viewed, causing the user interface to be displayed. In some embodiments, the power indicator provides a visual cue to inform the user of the direction in which head movement should be directed, causing the user interface to be displayed. In some embodiments, the display of the power indicator may stop after a condition is met. For example, the display of the power indicator may stop after a threshold duration has been displayed. As another example, the display of the power indicator may stop in response to a second user input corresponding to the second position or in response to the power indicator itself. In some embodiments, the display of the power indicator stops in response to a user request.

[0115] In some implementations, as illustrated in block 730c, the second user input includes head pose input. For example, head sensor data can be obtained from one or more head position sensors that sense the position and / or movement of the user's head. The one or more head position sensors may include, for example, an image sensor, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU). Head pose values ​​can be determined based on the head sensor data. The head pose values ​​may correspond to the orientation of the user's head toward a position within the field of view (e.g., the second user focus position).

[0116] In various embodiments, as indicated by box 740, method 700 includes displaying a first user interface under the condition that the second user focus position corresponds to a second position within the field of view that is different from the first position. In some embodiments, adjusting the activation of the first user interface upon receiving user input for different user focus positions reduces the incidence of false positives and unintentional activation of the first user interface during normal use. The amount of user input provided by the user can be reduced, for example, by reducing the amount of input required to correct false positives. Therefore, battery life can be enhanced. As indicated by box 740a, method 700 may include determining that the second user focus position corresponds to the second position under the condition that the second user focus position meets a proximity criterion relative to the second position. In some embodiments, as indicated by box 740b, electronic device 510 determines that the second user focus position corresponds to the second position if the second user focus position meets the proximity criterion for a duration of a threshold value.

[0117] In some implementations, as shown in block 740c, the electronic device 510 displays a first user interface under the condition that the first user input is maintained for a duration of a threshold duration and the second user focus position corresponds to the second position. For example, the user may be required to gaze at the first position for a duration of the threshold duration before gazing at the second position. Requiring the user to maintain a gaze at the first position reduces the incidence of false positives and reduces unintentional activation of the first user interface. The amount of user input provided by the user can be reduced, for example, by reducing the amount of input required to correct false positives. Therefore, battery life can be enhanced.

[0118] In some implementations, as indicated by box 740d, the visual properties of the first user interface are altered in response to the detection of third user input to the first user interface. The visual properties can be altered to increase or decrease the visibility of the first user interface. For example, as indicated by box 740e, the visual properties may include the color of the first user interface. The color of the first user interface may be altered to make the first user interface more visible or less visible relative to the background. In some implementations, as indicated by box 740f, the visual properties include the size of the first user interface. For example, the first user interface may be enlarged to make it more prominent. As another example, the first user interface may be reduced in size to make it less prominent.

[0119] Figure 8 This is a block diagram of a device 800 that activates the HUD interface using the first and second user focus positions, based on some specific implementations. In some specific implementations, device 800 implements... Figures 5A to 5H The electronic device 510 shown, and / or Figures 5A to 5H and Figure 6 The display interface engine 600 is shown. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring more relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, device 800 includes one or more processing units (CPUs) 801, a network interface 802, a programming interface 803, a memory 804, one or more input / output (I / O) devices 810, and one or more communication buses 805 for interconnecting these and various other components.

[0120] In some implementations, a network interface 802 is provided to establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices, among other uses. In some implementations, one or more communication buses 805 include circuitry for interconnecting and communicating between system components. Memory 804 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 804 optionally includes one or more storage devices remotely located to one or more CPUs 801. Memory 804 includes a non-transitory computer-readable storage medium.

[0121] In some embodiments, memory 804 or a non-transitory computer-readable storage medium of memory 804 stores programs, modules, and data structures, or subsets thereof, including optional operating system 806, environment renderer 610, image data acquirer 620, head pose value acquirer 630, and user interface generator 640. In various embodiments, device 800 executes... Figures 7A to 7B Method 700 is shown.

[0122] In some implementations, the environment renderer 610 displays an extended reality (XR) environment comprising a set of virtual objects within the field of view. In some implementations, the environment renderer 610 includes instructions 610a, as well as heuristics and metadata 610b.

[0123] In some specific implementations, the image data acquirer 620 acquires data from the captured user (e.g., Figure 5A The image data acquisition unit 620 acquires sensor data from one or more images of the user (512). In some embodiments, the image data acquisition unit 620 determines the gaze vector. In some embodiments, the image data acquisition unit 620 performs actions by... Figures 7A to 7B The operations represented by boxes 710, 720, and / or 730 are as follows. For this purpose, the image data acquirer 620 includes instructions 620a, as well as heuristics and metadata 620b.

[0124] In some implementations, the head pose value acquirer 630 acquires head sensor data from one or more head position sensors that sense the position and / or movement of the user 512's head. The one or more head position sensors may include, for example, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU). The head pose value acquirer 630 may generate head pose values ​​based on the head sensor data. In some implementations, the head pose value acquirer 630 performs actions by… Figures 7A to 7BThe operations represented by boxes 710, 720, and / or 730 in the diagram. For this purpose, the head pose value acquirer 630 includes instructions 630a, as well as heuristics and metadata 630b.

[0125] In some implementations, the user interface generator 640 causes, for example, the user interface to be displayed in an XR environment 514, provided that the user focus position 532 corresponds to the target position 546 and the user focus position 542 corresponds to the target position 536. In some implementations, the user interface generator 640 performs the actions of... Figures 7A to 7B Box 740 in the diagram represents the operation. For this purpose, the user interface generator 640 includes instructions 640a, as well as heuristics and metadata 640b.

[0126] In some specific implementations, one or more I / O devices 810 include a user-facing image sensor (e.g., Figure 6 One or more image sensors 622, which may be implemented as a forward-facing camera or an inward-facing camera. In some specific implementations, one or more I / O devices 810 include one or more head position sensors (e.g., for sensing the position and / or movement of the user's head). Figure 6 One or more head position sensors 634. The one or more head position sensors 634 may include, for example, an accelerometer, a gyroscope, a magnetometer, and / or an inertial measurement unit (IMU). In some embodiments, one or more I / O devices 810 include a display for displaying a graphical environment (e.g., for displaying an XR environment 514). In some embodiments, one or more I / O devices 810 include a speaker for outputting auditory signals.

[0127] In various embodiments, one or more I / O devices 810 include a video perspective display that displays at least a portion of the physical environment surrounding device 800 as an image captured by a scene camera. In various embodiments, one or more I / O devices 810 include an optical perspective display that is at least partially transparent and transmits light emitted or reflected by the physical environment.

[0128] It should be understood that Figure 8 This serves as a functional description of various features that may exist in a particular specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 8Some functional blocks shown individually can be implemented as a single block, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of blocks and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.

[0129] Figure 9 This is a flowchart illustrating a method 900 for displaying a user interface based on gaze and head movements, according to some specific implementations. In various implementations, method 900 comprises a device including one or more sensors, a display, one or more processors, and non-transitory memory (e.g., ...). Figure 1A The electronic device 100 shown is used for execution.

[0130] As illustrated in box 910, in various embodiments, method 900 includes receiving gaze data via one or more sensors that indicates a user's gaze pointing towards a location within the field of view. In some embodiments, receiving the gaze data includes utilizing an application programming interface (API) that provides the gaze data. For example, an application may make API calls to obtain the gaze data.

[0131] As illustrated in box 920, in various embodiments, method 900 includes receiving head pose data via one or more sensors that indicates head pose values ​​corresponding to a user's head pose. In some embodiments, receiving head pose data includes utilizing an API that provides head pose data. For example, an application may make API calls to obtain head pose data. In some embodiments, receiving head pose data includes receiving an indication of rotation of the head toward a position within the field of view. In some embodiments, rotation includes rotation of the head forward vector toward a position within the field of view. In some embodiments, head pose values ​​include a head pose vector that includes a set of one or more head pose values. In some embodiments, head pose values ​​include a single head pose value.

[0132] As indicated by box 930, in various specific embodiments, method 900 includes displaying a user interface on a display in response to head pose values ​​corresponding to movements of the user's head relative to a position in a predetermined manner. For example, when the user is gazing at a position and moving his / her head in a predetermined manner, the application may display the user interface 140 shown in Figure ID.

[0133] In some embodiments, method 900 includes displaying a visual indicator on the display near the location (e.g., at or near the location). In some embodiments, method 900 includes displaying a power indicator on the display near the location, which triggers the display of a user interface on the display (e.g., displaying power indicator 150 as shown in Figure IE) when activated by gazing at the power indicator and moving the head in a predetermined manner. In some embodiments, method 900 includes relinquishing the display of the user interface on the display in response to the user gazing at the pointed location and the user's head movement not being performed in a predetermined manner relative to the location. For example, user interface 140 as shown in Figure ID is not displayed when the user does not move his / her head in a predetermined manner, even if the user may be gazing at the location.

[0134] While various aspects of specific embodiments within the scope of the appended claims have been described above, it should be apparent that the various features of the above-described embodiments can be embodied in a wide variety of forms, and any particular structure and / or function described above are merely illustrative. Based on this disclosure, those skilled in the art will understand that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement an apparatus and / or practice a method. Furthermore, such an apparatus and / or such a method can be implemented using other structures and / or functions besides or different from one or more aspects set forth herein.

Claims

1. A method comprising: In devices that include sensors, displays, one or more processors, and memory: Within the field of view of the display, a first target position is determined corresponding to a first portion of the field of view, wherein the first portion is fixed relative to the field of view; After determining the location of the first target: Obtain a first user input corresponding to a first user focus position, wherein the first user input includes the user's gaze input; Based on the determination that the gaze indicated by the gaze input points to the first target location, it is determined that the first user focus position corresponds to the first target location; In response to determining that the first user focus position corresponds to the first target position, a visual effect associated with the first target position is displayed; Obtain a head pose value corresponding to the user's head starting to rotate toward the first target position, wherein the rotation is the user's initial head movement, the initial head movement including the user's head rotating toward the first target position after the visual effect is displayed, and wherein the gaze input and the head pose value are obtained independently of each other; Based on the combination of the gaze input and the head posture value obtained independently, it is determined that the activation conditions for displaying the first user interface are met, wherein the activation conditions include: the gaze indicated by the gaze input is directed toward the first target position, and the head posture value indicates that the rotation is directed toward the first target position during the initial head movement; as well as In response to determining that the activation condition is met, a first user interface is displayed.

2. The method of claim 1, wherein determining that the first user focus position corresponds to the first target position includes determining that the first user focus position satisfies a proximity criterion relative to the first target position.

3. The method of claim 2, wherein determining that the first user focus position corresponds to the first target position includes determining the duration for which the first user focus position satisfies the proximity criterion threshold.

4. The method of claim 1, further comprising displaying a first visual indicator near the first target location after determining the first target location and before obtaining the first user input.

5. The method of claim 4, further comprising stopping the display of the first visual indicator after the condition is met.

6. The method of claim 5, wherein stopping the display of the first visual indicator is performed in response to a threshold duration for displaying the first user interface.

7. The method of claim 5, wherein stopping the display of the first visual indicator is performed after the first user interface has been displayed a threshold number of times.

8. The method of claim 5, wherein stopping the display of the first visual indicator is performed in response to detecting the first user input for the first visual indicator.

9. The method of claim 1, further comprising changing visual attributes of the first user interface in response to detecting a second user input to the first user interface.

10. The method of claim 1, wherein displaying the first user interface is performed without determining that the gaze indicated by the gaze input has moved in the opposite direction to the movement of the user's head indicated by the head posture value during the initial head movement as a condition for displaying the first user interface.

11. The method of claim 1, wherein the gaze input indicates the user's gaze vector, and wherein the gaze indicated by the gaze input pointing to the first target location includes the gaze vector pointing to the first target location.

12. The method of claim 11, further comprising determining a second position associated with the gaze vector.

13. The method of claim 12, wherein determining that the first user focus position corresponds to the first target position includes determining that the second position associated with the gaze vector satisfies a proximity criterion relative to the first target position.

14. The method of claim 13, wherein determining that the first user focus position corresponds to the first target position includes determining that the second position associated with the gaze vector satisfies the proximity criterion duration threshold.

15. The method of claim 1, wherein the rotation of the user's head toward the first target position includes the rotation of the user's head forward vector toward the first target position, wherein the head forward vector indicates the direction in which the user's head is facing.

16. The method of claim 1, wherein obtaining the gaze input comprises capturing a set of one or more images of one or more eyes of the user by a user-facing image sensor, and determining a gaze vector indicative of the user's gaze based on the set of one or more images.

17. The method of claim 1, wherein obtaining the head pose value comprises acquiring head sensor data from one or more head position sensors that sense the position or movement of the user's head, and generating the head pose value based on the head sensor data.

18. The method of claim 1, wherein, Obtaining the head posture value includes acquiring sensor data from one or more of an image sensor, accelerometer, gyroscope, magnetometer, or inertial measurement unit.

19. An apparatus comprising: One or more processors; Non-transitory memory; monitor; Input devices; as well as One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: Within the field of view of the display, a first target position is determined corresponding to a first portion of the field of view, wherein the first portion is fixed relative to the field of view; After determining the location of the first target: Obtain a first user input corresponding to a first user focus position, wherein the first user input includes the user's gaze input; Based on the determination that the gaze indicated by the gaze input points to the first target location, it is determined that the first user focus position corresponds to the first target location; In response to determining that the first user focus position corresponds to the first target position, a visual effect associated with the first target position is displayed; Obtain a head pose value corresponding to the user's head starting to rotate toward the first target position, wherein the rotation is the user's initial head movement, the initial head movement including the user's head rotating toward the first target position after the visual effect is displayed, and wherein the gaze input and the head pose value are obtained independently of each other; Based on the combination of the gaze input and the head posture value obtained independently, it is determined that the activation conditions for displaying the first user interface are met, wherein the activation conditions include: the gaze indicated by the gaze input is directed toward the first target position, and the head posture value indicates that the rotation is directed toward the first target position during the initial head movement; as well as In response to determining that the activation condition is met, a first user interface is displayed.

20. A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device, cause the device to: Within the field of view of the display, a first target position is determined corresponding to a first portion of the field of view, wherein the first portion is fixed relative to the field of view; After determining the location of the first target: Obtain a first user input corresponding to a first user focus position, wherein the first user input includes the user's gaze input; Based on the determination that the gaze indicated by the gaze input points to the first target location, it is determined that the first user focus position corresponds to the first target location; In response to determining that the first user focus position corresponds to the first target position, a visual effect associated with the first target position is displayed; Obtain a head pose value corresponding to the user's head starting to rotate toward the first target position, wherein the rotation is the user's initial head movement, the initial head movement including the user's head rotating toward the first target position after the visual effect is displayed, and wherein the gaze input and the head pose value are obtained independently of each other; Based on the combination of the gaze input and the head posture value obtained independently, it is determined that the activation conditions for displaying the first user interface are met, wherein the activation conditions include: the gaze indicated by the gaze input is directed toward the first target position, and the head posture value indicates that the rotation is directed toward the first target position during the initial head movement; as well as In response to determining that the activation condition is met, a first user interface is displayed.

Citation Information

Patent Citations

  • Method, apparatus, and system for modeling interactions of a group of users with a computing system

    US20130311925A1

  • Radial selection by vestibulo-ocular reflex fixation

    US20150212576A1