Gaze-based control for electronic devices
The gaze tracker and processor detect user gaze movement, and control electronic device functions using the movement and residence time of the activation area, solve the problem that conventional input devices are not feasible, improve the user experience, especially suitable for hands occupied or head-wearing scene of equipment.
Patent Information
- Application Number
- CN202410673541.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-16
- Filing Date
- 2024-05-28
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, when using conventional user input mechanisms (such as buttons and touch-sensitive screens) to control electronic devices, it may lead to reduced or unfeasible user experience, especially when the user's hands are occupied or the head-mounted device is .
The gaze tracker and processor are employed to detect the user's gaze movement, activate the functions of the electronic device by defining the movement and residence time of the activation area within the gaze field of view, and provide feedback signals to help the user activate or avoid accidental activation.
It realizes that the user experience is improved, especially when the hands are occupied or head-mounted devices are controlled by gaze without using traditional input devices.
Smart Images

Figure CN119065492A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is a non-provisional application of U.S. Provisional Patent Application No. 63 / 470,033 filed on May 31, 2023, and claims the benefit of the U.S. Provisional Patent Application No. 63 / 470,033 under 35 U.S.C. 119(e), the contents of which are incorporated herein by reference as if fully disclosed herein. Technical Field
[0003] Embodiments described herein relate to systems and methods for controlling an electronic device using a user's gaze. Background Art
[0004] An electronic device may be configured to perform one or more functions in response to user input at one or more conventional user input mechanisms (such as buttons and touch-sensitive screens). However, there may be situations where user input via conventional user input mechanisms degrades the quality of the user experience or is otherwise not feasible or desirable. Summary of the invention
[0005] Embodiments described herein relate to systems and methods for controlling an electronic device using a user's gaze. In one embodiment, the electronic device may include a gaze tracker, a display, and a processor. The gaze tracker may be configured to detect a user's gaze within a gaze field of view. The display may have a display area positioned to overlap a portion of the gaze field of view and may be contained within the gaze field of view. The processor may be operably coupled to the gaze tracker and the display. The processor may be configured to detect movement of the user's gaze to an activation area of the gaze field of view outside the display area, and in response, activate a function of the electronic device.
[0006] In one embodiment, the electronic device includes a frame including a gaze tracker and a display. The frame can define a frame area positioned to overlap the gaze field and include a display area. The frame area can be within the gaze field, and the activation area can be outside the frame area.
[0007] In one embodiment, the processor is configured to cause a feedback signal to be provided to the user in response to detecting movement of the user's gaze toward the activation area. The intensity of the feedback signal may increase relative to the amount of time the user's gaze remains in the activation area and / or in response to the proximity of the user's gaze to the activation area. The feedback signal may be light provided by a light source positioned in the frame near the activation area. The feedback signal may be a tactile signal and / or an audio signal.
[0008] In one embodiment, the activation area may be located in a corner of the fixation field of view. The activation area may additionally or alternatively abut an edge of the fixation field of view.
[0009] In one embodiment, unless the user's gaze remains in the activation area for at least the dwell time, the function may not be activated.
[0010] In one embodiment, an electronic device includes a gaze tracker and a processor operably coupled to the gaze tracker. The gaze tracker can be configured to detect a gaze of a user within a gaze field of view. The processor can be configured to detect movement of the user's gaze to an activation area in the gaze field of view; determine one or more characteristics of the movement of the user's gaze to the activation area; determine whether the user's gaze remains in the activation area for at least a dwell time, the dwell time based on one or more characteristics of the movement of the user's gaze; and in response, activate a function of the electronic device.
[0011] In one embodiment, one or more characteristics of the movement of the user's gaze may include the speed of the movement. The dwell time may be inversely proportional to the speed of the movement of the user's gaze.
[0012] In one embodiment, one or more characteristics of the movement of the user's gaze may include a path shape of the movement of the user's gaze from the starting point to the activation area. The dwell time may be based at least in part on the relationship between the path shape of the movement of the user's gaze and the boundary of the activation area.
[0013] In one embodiment, the electronic device may also include a display having a display area positioned to overlap the gaze field. The display area may be contained within the gaze field. The activation area may be outside the display area.
[0014] In one embodiment, the electronic device may further include a frame including a gaze tracker and a display. The frame may define a frame area positioned to overlap the gaze field of view. The display area may be contained within the frame area. The frame area may be contained within the gaze field of view.
[0015] In one embodiment, the activation area may be located in a corner of the fixation field of view. The activation area may additionally or alternatively abut an edge of the fixation field of view.
[0016] In one embodiment, an electronic device includes one or more sensors, a gaze tracker, and a processor. The gaze tracker may be configured to detect a gaze of a user within a gaze field of view. The processor may be operably coupled to the one or more sensors and the gaze tracker. The processor may be configured to detect movement of the user's gaze to an activation area of the gaze field of view; modify the activation area based on one or more sensor signals from the one or more sensors; and activate a function of the electronic device in response to detecting movement of the user's gaze to the modified activation area.
[0017] In one embodiment, modifying the activation region includes enabling and disabling the activation region. When the activation region is disabled, the function may not be activated.
[0018] In one embodiment, modifying the activation area may include changing the size of the activation area.
[0019] In one embodiment, the activation area is associated with a function of the electronic device, and modifying the activation area is based on the function of the electronic device associated with the activation area.
[0020] In one embodiment, the one or more sensor signals indicate movement of the electronic device.
[0021] In one embodiment, the one or more sensors may include a camera having an imaging field of view that at least partially overlaps the gaze field of view and includes the activation area. One or more sensor signals may indicate whether the camera detects the presence of an object that matches object criteria in the activation area. The object criteria may include the type of the object, the size of the object, and the proximity of the object to the electronic device.
[0022] In one embodiment, the one or more sensor signals indicate that the user is engaged in an activity matching the activity criteria.
[0023] In one embodiment, the electronic device may further include a frame including a gaze tracker and a display. The frame may define a frame area positioned to overlap the gaze field of view. The display area may be contained within the frame area. The frame area may be contained within the gaze field of view.
[0024] In one embodiment, the activation area may be located in a corner of the fixation field of view. The activation area may additionally or alternatively abut an edge of the fixation field of view. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Reference will now be made to the representative embodiments illustrated in the accompanying drawings. It should be understood that the following description is not intended to limit the present disclosure to an included embodiment. On the contrary, the disclosure provided by the present disclosure is intended to cover alternatives, modifications and equivalents that may be included within the spirit and scope of the embodiments and defined by the appended claims.
[0026] Figure 1 A block diagram of an electronic device such as described herein is depicted.
[0027] Figure 2A-2I Depicted are diagrams showing portions of a physical environment from the perspective of an electronic device such as described herein.
[0028] Figure 3 is a flow chart depicting exemplary operations of a method of controlling an electronic device using a user's gaze, such as described herein.
[0029] Figure 4 is a flow chart depicting exemplary operations of a method of controlling an electronic device using a user's gaze, such as described herein.
[0030] Figure 5 is a flow chart depicting exemplary operations of a method of controlling an electronic device using a user's gaze, such as described herein.
[0031] The use of the same or similar reference numbers in different drawings indicates similar, related or identical items.
[0032] The use of cross-hatching or shading in the drawings is generally provided to clarify boundaries between adjacent elements and also to facilitate legibility of the drawings. Thus, the presence or absence of cross-hatching or shading does not express or indicate any preference or requirement for particular materials, material properties, element proportions, element sizes, commonality of similar illustrated elements, or any other feature, attribute, or characteristic of any element shown in the drawings.
[0033] Additionally, it should be understood that the proportions and sizes (relative or absolute) of the various features and elements (and collections and groupings thereof) and the boundaries, spacings, and positional relationships presented therebetween are provided in the accompanying drawings solely to facilitate understanding of the various embodiments described herein, and therefore may not necessarily be presented or illustrated to scale and are not intended to indicate any preference or requirement for the embodiments illustrated to exclude the embodiments described in conjunction therewith. DETAILED DESCRIPTION
[0034] Embodiments described herein relate to systems and methods for controlling electronic devices using the user's gaze. In some cases, it may be difficult, impractical, or undesirable to control an electronic device using conventional user input mechanisms (such as buttons and touch-sensitive screens). For example, in the case where the user's hands are occupied or the electronic device is a head-mounted device, interacting with the electronic device in a traditional manner may reduce the user experience. The system and method of the present application are configured to control an electronic device using the user's gaze.
[0035] In one example, a user may direct their gaze to an activation area within a gaze field to activate a function of an electronic device. An activation area may be an area within a portion of a user's gaze (e.g., a gaze field) that a user can look at to activate a function of an electronic device. For example, an activation area may be a corner of a gaze field or an edge of a gaze field. For example, a user may direct their gaze to an activation area to display a graphical user interface at a display. As another example, a user may direct their gaze to an activation area to answer an incoming phone call. A gaze field may include multiple activation areas, each associated with a different function of an electronic device. Thus, a user's gaze may be used to control an electronic device.
[0036] In some cases, feedback may be provided to the user to help activate the function associated with the activation area. The feedback may be audio feedback, visual feedback, tactile feedback, a combination thereof, etc. In some cases, the intensity of the feedback may increase relative to the proximity of the user's gaze to the activation area and / or the amount of time the user's gaze remains within the activation area. In one example, the electronic device includes a frame and a light source positioned in the frame near the activation area. Light may be provided from the light source to indicate when the user's gaze is near or within the activation area. In addition, the intensity of the light may increase relative to the proximity of the user's gaze to the activation area and / or the amount of time the user's gaze remains within the activation area. The feedback may indicate that the function is about to be activated and give the user the opportunity to move his gaze or otherwise perform an action to prevent the function from being activated.
[0037] In some cases, the function of the electronic device can be activated only when the user's gaze remains in the activation area for at least the dwell time. This can prevent the user from accidentally activating the function. The dwell time can be predefined or dynamically defined. Specifically, the dwell time can be dynamically defined based on one or more characteristics of the user's gaze moving to the activation area. For example, the speed at which the user's gaze moves to the activation area can be used to determine the dwell time, because the speed can indicate the user's intention to activate the function. The faster movement of the user's gaze to the activation area can indicate that the user intends to activate the function with greater confidence than the slower movement. Therefore, the speed at which the user's gaze moves to the activation area can be inversely proportional to the dwell time. As another example, the path shape of the user's gaze moving to the activation area can be used to determine the dwell time, because the path can indicate the user's intention to activate the function. The direct path between the initial point of the user's gaze and the activation area can indicate that the user intends to activate the function with greater confidence than the indirect path. Therefore, the direction of the path shape of the movement of the user's gaze can be inversely proportional to the dwell time.
[0038] In some cases, the activation area may be modified in response to the environment or other information determined by the electronic device. Specifically, the size of the activation area may be increased or decreased, the shape of the activation area may be modified, the dwell time in the activation area required to activate the function may be increased or decreased, and / or the activation area may be enabled or disabled to increase or decrease the sensitivity of the activation area used to activate the function. For example, when a user is performing certain activities (such as running), the risk of the user accidentally looking at the activation area and / or generating a false positive due to inaccuracy of the gaze tracking hardware may be higher than usual. Therefore, when it is determined that the user is performing certain activities, the area of the activation area may be reduced, the dwell time required to activate the function may be increased, and / or the activation area may be disabled. On the contrary, it may be desirable to make it easier to activate the function during certain activities, and therefore when it is determined that the user is performing certain activities, the area of the activation area may be increased, the dwell time required to activate the function may be reduced, and / or the activation area may be enabled.
[0039] As another example, the activation area may be modified based on the function associated with the activation area. It may be desirable to more aggressively prevent accidental activation of certain functions of the electronic device (such as answering an incoming phone call or unlocking the front door) than other functions (such as displaying a graphical user interface or silencing an incoming phone call). For functions where the user wishes to more aggressively avoid accidental activation, the area of the activation area may be reduced and / or the dwell time required to activate the function may be increased.
[0040] As yet another example, some types of objects may be located in the activation area. For example, a person may be located in the gaze field within the activation area. A user may look at a person in the activation area but not intend to activate the function. Thus, when a person or other object is detected in the activation area, the activation area may be disabled, or the activation area may be modified to exclude the area occupied by the person or other object.
[0041] The following references Figures 1 to 5 These foregoing embodiments and other embodiments are discussed. However, those skilled in the art will readily appreciate that the detailed description given herein with respect to these drawings is for explanation only and should not be construed as limiting.
[0042] Figure 1 1 is a simplified block diagram of an electronic device 100 according to one embodiment of the present disclosure. The electronic device 100 includes a processor 102, a memory 104, a plurality of sensors 106 including at least a gaze tracker 106a, a display 108, one or more cameras 110, and one or more feedback mechanisms 112 (e.g., one or more light sources 112a, one or more speakers 112b, one or more tactile actuators 112c, combinations thereof, etc.). The framework 114 may support all or part of the processor 102, the memory 104, the sensor 106, the display 108, the one or more cameras 110, and the one or more feedback mechanisms 112. The memory 104, the sensor 106, the display 108, the one or more cameras 110, and the one or more feedback mechanisms 112 may be operably coupled to the processor 102. The memory 104 may include instructions that, when executed by the processor 102, cause the electronic device 100 to perform the operations described herein to control the electronic device via the gaze of the user. The gaze tracker 106a may be configured to detect the position / direction of the user's gaze within the gaze field of view. The gaze tracker 106a may include any suitable hardware for tracking the gaze of the user, such as one or more cameras, depth sensors, combinations thereof, etc. The plurality of sensors 106 may include any number and type of sensors, such as motion sensors (e.g., accelerometers, gyroscopes), light sensors, biosensors (e.g., heart rate sensors, breathing sensors, etc.), or any other type of sensor. In some embodiments, the electronic device 100 may be a head-mounted device, such as an extended reality headset, smart glasses, etc. However, the principles of the present disclosure are applicable to electronic devices having any form factor.
[0043] Figure 2A It shows that the above Figure 1A diagram of an exemplary relationship between the object detection field of view 200, the gaze field of view 202, and the display area 204 of the electronic device 100. The object detection field of view 200 may correspond to the boundary of the environment in which the electronic device 100 can perceive and / or identify objects. For example, the object detection field of view 200 may correspond to the field of view of one or more cameras 110 of the electronic device 100, or a combination of the field of view of one or more cameras 110 and one or more other sensors (such as one or more depth sensors, one or more lidar sensors) or any other sensor for detecting and / or identifying objects in the physical environment. The gaze field of view 202 may correspond to an area on which the user's gaze can be tracked. In some embodiments, this may correspond to an area in which the position of the user's gaze can be determined with the desired accuracy. The gaze field of view 202 may represent all or a subset of the full gaze range of the user for a given head position. As shown, the gaze field of view 202 may be a subset of the object detection field of view 200, however, this is not required. In some cases, the gaze field of view 202 may be the same size as or larger than the object detection field of view 200. Display area 204 may correspond to the area on which a graphical user interface is superimposed when viewed by a user of electronic device 100 (e.g., when electronic device 100 is worn as a head mounted device). In some cases, display area 204 may correspond to the physical boundaries of display 108 of electronic device 100. In some cases, display area 108 overlaps at least partially with object detection field of view 200 and gaze field of view 202, and as shown, display area 204 may be smaller than both object detection field of view 200 and gaze field of view 202. However, this is not required, and display area 204 may be the same or larger than the size of object detection field of view 200 and / or gaze field of view 202. Display 108 may be at least partially transparent so that the physical environment is visible through display area 204, and graphical elements may be superimposed on the physical environment within display area 204.
[0044] Figure 2B The object detection field of view 200, the gaze field of view 202, and the display area 204 are shown as they relate to the user 208 of the electronic device 100. For context, the gaze tracker 106a, the display 108, and the one or more cameras 110 are also shown. As shown, the object detection field of view 200 may correspond to the field of view of the one or more cameras 110. The gaze field of view 202 may correspond to the area in which the user's gaze can be detected by the gaze tracker 106a. This can be limited based on the constraints of the gaze tracker 106a (such as its position, its capabilities, etc.). The display area 204 corresponds to the position of the display 108 relative to the user. Although not shown, the display 108 can be supported by the frame 114 at a specific distance from the user 208 and at a specific position relative to the user 208. This can determine the display area 204 observed by the user 208.
[0045] Figure 2C An object detection field of view 200, a gaze field of view 202, and a display area 204 are shown in association with a frame 114 of an electronic device 100. As shown, the frame 114 physically supports a gaze tracker 106a and one or more cameras 110. In addition, the frame 114 may physically support one or more feedback mechanisms 112, which in this example are shown as light sources embedded in the frame 114. The one or more feedback mechanisms 112 may be distributed in the frame near one or more activation areas in the gaze field of view, as discussed in further detail below. The frame 114 defines a frame area 210, which may correspond to an area through which a user views when operating the electronic device 100. The frame area 210 may include at least a portion of the gaze field of view 202 and the display area 204. As described above, the electronic device 100 may be a head-mounted device. The frame 114 may correspond to a housing of the electronic device. In one embodiment, the frame 114 has the form factor of an eyeglass frame and is configured to be a plurality of substantially identical portions of the display area 204. Figure 2C 1 and 10. One side of the frame 114 (one eyepiece) is shown in FIG. 1. Although the feedback mechanism 112 is shown as a light source in this example, the feedback mechanism 112 may include a speaker, a tactile actuator, or other types of feedback mechanisms 112. Some types of feedback mechanisms may not make sense to place near the activation area and may be placed in other locations, such as in an earpiece of the electronic device 100.
[0046] Figure 2D A gaze field of view 202 including a plurality of activation regions 212 is shown. For context, a display region 204 is also shown, wherein the activation region 212 is outside the display region 204. Providing the activation region 212 outside the display region 204 allows a wider range of the gaze field of view 202 to be used to activate the functions of the electronic device 100. That is, providing the activation region 212 outside the display region 204 ensures that the area for activating the functions of the electronic device 100 is not limited to the display region 204. However, in some embodiments, the activation region 212 may overlap partially or completely with the display region 204. The activation region 212 may be located in each corner of the gaze field of view 202 and between the corners at the edge of the gaze field of view 202. It is worth noting that the size and position of the activation region 212 may vary depending on its specific implementation. For example, the gaze field of view 202 may include only a subset of the activation regions 212 shown, or may include additional activation regions 212 not shown. In addition, the shape and / or position of the activation region 212 may be different, but without departing from the principles of the present disclosure.
[0047] The activation areas 212 can be configured by the user of the electronic device 100. For example, the user can activate or deactivate certain activation areas 212, change the size or shape of any of the activation areas 212, and specify functions associated with specific activation areas. As discussed above, the activation areas 212 are areas in the gaze field of view 202 where the user can direct their gaze to activate functions of the electronic device 100. Each activation area 212 can be associated with a function of the electronic device 100 so that when the user directs their gaze to the activation area 212, the associated function is activated. Different activation areas 212 can be associated with different functions. For example, a first activation area 212 can be associated with a first function for showing a settings graphical user interface in the display area 204, and a second activation area 212 can be associated with a second function for activating a smart home device (such as a smart light bulb).
[0048] Figure 2E The movement of the user's gaze from a first position 214a to a second position 214b is shown, wherein the first position 214a is not within the activation area 212 and the second position 214b is within the activation area 212. The electronic device 100 may be configured to detect (e.g., using the gaze tracker 106a) the movement of the user's gaze to the activation area 212 (or any other activation area 212), and in response thereto, activate a function of the electronic device 100. For example, in response to detecting the movement of the user's gaze to the activation area 212, the electronic device 100 may display a graphical user interface (such as a settings graphical user interface) in the display area 204. As another example, in response to detecting the movement of the user's gaze to the activation area 212, the electronic device 100 may answer an incoming phone call, activate a voice command or a virtual assistant, open an application, or perform a custom device operation specified by the user. In some cases, the user's gaze may be required to stay in the activation area 212 for at least a dwell time before activating the function. In some cases, this can reduce accidental triggering of functions associated with the activation area 212.
[0049] The movement of the user's gaze may have associated characteristics, which may be measured by the electronic device 100 via the gaze tracker 106a or otherwise. For example, the movement of the user's gaze may have an associated speed of movement between the first position 214a and the second position 214b. As another example, the movement of the user's gaze may track the path 216 between the first position 214a and the second position 214b. These characteristics may indicate the user's intention to activate the function associated with the activation area 212. For example, a higher speed of movement of the user's gaze to the second position 214b may indicate with a higher confidence than a lower speed that the user intends to activate the function associated with the activation area 212. As another example, the movement of the user's gaze along a direct path between the first position 214a and the second position 214b may indicate with a higher confidence than a less direct path that the user intends to activate the function associated with the activation area 212. These characteristics may be used to modify the characteristics of one or more activation areas 212 to make it more or less difficult to activate the function associated with it. For example, characteristics of the movement of the user's gaze may be used to determine the dwell time required in the activation area 212 before activating the function.
[0050] When the characteristic indicates a higher confidence that the user intends to activate the function, the dwell time can be reduced. Conversely, when the characteristic indicates a lower confidence that the user intends to activate the function, the dwell time can be increased. In various examples, the dwell time can be inversely proportional to the speed at which the user's gaze moves toward the activation area 212, and inversely proportional to the path direction of the user's gaze moving toward the activation area 212. In some embodiments, the relationship between the path shape of the user's gaze moving toward the activation area 212 and the boundary of the activation area 212 can be used to determine the dwell time. For example, if the shape of the path of movement is largely parallel to the boundary of the activation area 212, this may indicate a lower confidence that the user intends to activate the function, and the dwell time can be increased.
[0051] In some cases, it may be desirable to provide feedback to the user before activating a function associated with activation region 212 or otherwise guide the user to interact with activation region 212. For example, one or more feedback signals, such as audio signals, visual signals, tactile signals, or any other type of feedback signal, may be provided to the user before activating a function. The feedback may give the user an opportunity to prevent the function from activating, for example, by moving their gaze away from activation region 212. Figure 2FAn exemplary feedback provided to a user having their gaze in an activation area 212 via a feedback mechanism 112 in a frame 114 is shown. As shown, the feedback mechanism 112 is a light source embedded in the frame 114 near the activation area 212 in the gaze field of view 202. A light source near the activation area 212 where the user's gaze is located (as indicated by a second point 214b in the activation area 212) provides light. This may indicate that the user's gaze is within the activation area 212 and that the function associated with the activation area 212 will be activated unless the user moves their gaze or takes other action. In some cases, the intensity of the feedback provided to the user may increase in relation to (e.g., proportionally) the amount of time the user's gaze remains within the activation area 212 or in relation to (e.g., proportionally) the proximity of the user's gaze to the activation area 212.
[0052] For example, the closer the user's gaze is to the activation area 212, or the longer the user's gaze remains within the activation area 212, the intensity of the light provided by the light source may increase. In one example, feedback having a first intensity level is provided when the user's gaze approaches the activation area 212, and increases to a second intensity level when the user's gaze enters the activation area 212. When the user's gaze remains within the activation area 212, the feedback may increase to a third intensity level. Thus, the intensity of the feedback may indicate the immediacy with which a function associated with the activation area 212 will be activated, thereby giving the user an opportunity to avoid activating the function if desired.
[0053] In addition to modifying the dwell time required to activate a function associated with activation region 212, other characteristics of activation region 212 may be modified to make it more or less difficult for a user to activate the function and thereby avoid accidental activation. Figure 2GAs shown, the size of the activation area 212 can be increased or decreased, wherein the original activation area 212 is replaced by an updated activation area 242 having a smaller size. Although not shown, the shape of the activation area 212 can also be changed to make it more or less difficult for the user to activate the function. As the activation area 212 becomes smaller and smaller, it becomes more difficult for the user to activate the function because the area to which the user must direct their gaze is smaller. Therefore, in the case of a high risk of accidental activation, such as when the position of the user's gaze cannot be determined with the desired accuracy (for example, when there is a large amount of movement of the electronic device 100, such as when the user is exercising), in the case of a significant amount of expected user's gaze movement (for example, when sightseeing or hiking), or in any other case, the size of the activation area 212 and / or the shape of the activation area 212 can be modified to make it more difficult to activate the function associated with it. In some cases where the risk of accidental activation is too high, the activation area 212 can be completely disabled so that even when the user's gaze is directed to the activation area 212 and / or stays on the activation area 212, the function is not activated.
[0054] Characteristics of the activation area 212, such as size, shape, enablement, or any other characteristic including dwell time, may be modified based on measurements from one or more of the sensors 106 (e.g., sensor signals, such as motion sensor signals from an accelerometer and / or a gyroscope). This may include a gaze tracker 106a and therefore characteristics of the movement of the user's gaze as described above. Measurements from the sensors 106 may indicate the environment of the user and / or the electronic device 100, such as whether the user is exercising, whether the user is in a situation where there is a high risk of accidental activation, whether the user is in a situation where certain functions are inappropriate (e.g., if the function is playing music through a speaker and the user is in a work meeting), etc. In one example, the electronic device 100 may determine that the user is looking at the screen of the electronic device. Since the user may be looking at any part of the screen, any activation area 212 that overlaps with the screen may be at a high risk of accidental activation. Therefore, one or more activation areas 212 may be modified to exclude the area occupied by the screen.
[0055] In general, any sensor and therefore any measurement result may be used to determine when it is appropriate to modify the activation area 212 to increase or reduce the difficulty of activating the associated function. In one example, a sensor signal from one or more sensors may indicate that the user is engaged in an activity matching the activity criteria (e.g., the user is exercising as described above). If the user is engaged in an activity matching the activity criteria, the activation area 212 may be modified. Although the sensor signal from one or more sensors may be used to determine the environment of the user and / or the electronic device 100 for modifying one or more activation areas 212, additional information may be used alternatively or additionally for the same purpose. For example, user input to the electronic device 100 or another electronic device (such as a smart watch or smart phone) may be used to determine the environment of the user and / or the electronic device 100, and thus modify one or more activation areas 212. For example, a user may interact with the electronic device 100 or another electronic device to start an exercise in an exercise tracking application, which may indicate that there is an increased risk of accidental activation of one or more activation areas 212 as described above. As another example, the user may be playing a game on another device, which may also indicate that there is an increased risk of accidental activation of one or more activation areas 212. In general, any information collected directly or indirectly by the electronic device 100 may be used to determine when it is appropriate to modify one or more of the activation regions 212 .
[0056] In some cases, the function associated with the activation area 212 can be used to determine one or more characteristics thereof, such as size, shape, enablement, etc. Certain functions may be associated with an increased expectation of avoiding accidental activation compared to other functions. For example, answering an incoming phone call may be associated with an increased expectation of avoiding accidental activation compared to muting an incoming phone call or activating a virtual assistant. These preferences may be indicated by the user or determined by device default. The size, shape, or other aspects of an activation area 212 associated with a function (which in turn is associated with an increased expectation of avoiding accidental activation) may be modified to make it more difficult to activate the function than other activation areas 212, which are associated with functions that are not associated with an increased expectation of avoiding accidental activation. In addition, these activation areas 212 associated with functions (which in turn are associated with an increased expectation of avoiding accidental activation) may be disabled at a lower risk threshold of accidental activation than other activation areas 212.
[0057] In some cases, such as Figure 2H As shown, there may be an object 218 that overlaps with the activation area 212 that the user may be looking at. The user may look at the object 218 without intending to activate the function associated with the activation area 212. For example, the object 218 may be a person located in the activation area 212. The user may wish to look at this person, such as to have a conversation, without activating the function associated with the activation area 212. Therefore, Fig.2I Modification of activation area 212 is shown to exclude object 218 so that a user can look at object 218 without activating functionality associated with activation area 212. Object 218 may be detected by one or more cameras 110 or any other sensor 106 of electronic device 100. In some cases, only objects that match object criteria will result in modification of activation area 212. Object criteria may include the type of object, the size of the object, the proximity of the object to electronic device 100, or any other criteria. Changes (e.g., size, shape, dwell time) to activation area 212 may depend on the type, size, proximity, or any other characteristic of the object detected in activation area 212.
[0058] Figure 3 3 is a flowchart depicting exemplary operations of a method 300 for activating a function of an electronic device using a user's gaze. These operations may be, for example, as described above with respect to Figure 1 The electronic device 100 discussed is performed. At box 302, the movement of the user's gaze to the activation area of the gaze field can be detected. The movement of the user's gaze can be detected, for example, by one or more sensors (such as a gaze tracker). At box 304, feedback related to the relationship between the user's gaze and the activation area can be provided to the user. For example, feedback can be provided to the user to indicate that their gaze has entered the activation area, indicate the proximity of their gaze to the activation area, or indicate that the function associated with the activation area is about to be activated. The feedback can be audio, visual, tactile or any other type of feedback. In some embodiments, the intensity of the feedback is provided in proportion to the amount of time the user's gaze remains in the activation area, the proximity of the user's gaze to the activation area, the immediacy of the activation of the function associated with the activation area, or any other information.
[0059] At block 306, it may be determined whether the user's gaze remains in the activation area for an amount of time greater than or equal to the dwell time. As described above, this may reduce the risk of accidentally activating a function associated with the activation area. If the user's gaze remains in the activation area for at least the dwell time, or if block 306 is omitted, the function is activated at block 308. For example, a graphical user interface may be displayed at a display of the electronic device, the electronic device may communicate with the smart home device to activate the smart home device, or any other function may be performed.
[0060] Figure 4 4 is a flowchart depicting exemplary operations of a method 400 for activating a function of an electronic device using a user's gaze. These operations may be, for example, as described above with respect to Figure 1The electronic device 100 discussed may be performed. At block 402, movement of a user's gaze to an activation area of a gaze field may be detected. The movement of the user's gaze may be detected, for example, by one or more sensors (such as a gaze tracker). At block 404, feedback may be provided to the user regarding the relationship of the user's gaze to the activation area, as described above.
[0061] At block 406, one or more characteristics of the movement of the user's gaze are determined. The one or more characteristics may include any characteristics such as, for example, the speed of the movement and the shape of the path of the movement from the initial point to the activation area. As described above, different characteristics of the movement of the user's gaze may indicate the confidence that the user intends to activate the function associated with the activation area.
[0062] At block 408, a dwell time is determined based on one or more characteristics of the movement of the user's gaze. For example, when the one or more characteristics indicate a lower confidence that the user intends to activate a function associated with the activation area, the dwell time can be increased, and conversely, when the one or more characteristics indicate a higher confidence that the user intends to activate a function associated with the activation area, the dwell time can be decreased.
[0063] At block 410, it is determined whether the user's gaze remains within the activation zone for an amount of time greater than or equal to the dwell time. If the user's gaze remains within the activation zone for at least the dwell time, then the function is activated at block 412, as described above.
[0064] Figure 5 is a flow chart depicting exemplary operations of a method 500 for activating a function of an electronic device using a user's gaze. These operations may be, for example, as described above with respect to Figure 1 The electronic device 100 discussed herein may be performed. At block 502, movement of a user's gaze to an activation area of a gaze field may be detected. Movement of the user's gaze may be detected, for example, by one or more sensors, such as a gaze tracker. At block 504, feedback may be provided to the user regarding the relationship of the user's gaze to the activation area, as described above.
[0065] At block 506, one or more sensor signals are received from one or more sensors. The one or more sensor signals may represent physical phenomena in the physical environment, such as movement of a user and / or electronic device, presence or absence of an object in front of the user and / or electronic device, or any other information.
[0066] At block 508, the activation area is modified based on the one or more sensor signals. For example, the size of the activation area, the shape of the activation area, the dwell time associated with the activation area, whether the activation area is enabled or disabled, or any other characteristic of the activation area can be modified in response to the one or more sensor signals. As described above, the one or more sensor signals can indicate an increased risk of accidental activation of a function associated with the activation area. Modifying the characteristics of the activation area based on the sensor signals can thus reduce the likelihood of accidentally triggering a function of the electronic device.
[0067] At block 510, it may be determined whether the user's gaze remains in the activation region for an amount of time greater than or equal to the dwell time. If the user's gaze remains in the activation region for at least the dwell time, or if block 510 is omitted, the function is activated at block 512, as described above.
[0068] Figures 1 to 5 These foregoing embodiments and various alternatives and variations thereof depicted in the foregoing are generally presented for purposes of explanation and to facilitate understanding of various configurations and constructions of systems such as those described herein. However, it will be apparent to those skilled in the art that some of the specific details presented herein may not be required to practice a particular described embodiment or its equivalent.
[0069] Therefore, it should be understood that the above and following descriptions of specific embodiments are presented for the specific purpose of illustration and description. These descriptions are not intended to be exhaustive or to limit the present disclosure to the precise forms described herein. On the contrary, it will be apparent to those of ordinary skill in the art that many modifications and variations are possible in light of the above teachings.
[0070] As used herein, the phrase "at least one of" following a list of items, any of which is separated by the terms "and" or "or," modifies the list as a whole, rather than each member of the list. The phrase "at least one of" does not require selection of at least one of each of the listed items; rather, the phrase allows for a meaning that includes at least one of any of the items and / or at least one of any combination of the items and / or at least one of each of the items. For example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" each refers to only A, only B, or only C; any combination of A, B, and C; and / or one or more of each of A, B, and C. Similarly, it should be understood that the order in which the elements are presented for the combined or separated lists provided herein should not be construed to limit the disclosure to only the order provided.
[0071] It is to be understood that, although many embodiments are disclosed above, the operations and steps provided with respect to the methods and techniques described herein are intended to be exemplary and therefore not exhaustive. It is further understood that alternative step sequences or fewer or additional operations may be required or desired for specific embodiments.
[0072] Although the above disclosure is described in terms of various exemplary embodiments and specific implementations, it should be understood that the various features, aspects, and functions described in one or more individual embodiments are not limited to their application to the specific embodiments in which they are described, but rather they may be applied alone or in various combinations to one or more of some embodiments of the present invention, regardless of whether such embodiments are described and whether such features are presented as part of the embodiments. Therefore, the breadth and scope of the present invention should not be limited by any of the above exemplary embodiments, but instead by the claims provided herein.
[0073] As used herein, the term "processor" refers to any software and / or hardware-implemented data processing device or circuitry that is physically and / or structurally configured to exemplify one or more classes or objects that are specifically configured to perform specific transformations of data, including operations represented as code and / or instructions included in a program that can be stored in and accessed from a memory. The term is intended to encompass a single processor or processing unit, multiple processors, multiple processing units, analog or digital circuits, or other appropriately configured computing elements or combinations of elements.
[0074] The present disclosure also includes the following embodiments:
[0075] Clause 1. An electronic device, comprising:
[0076] a gaze tracker configured to detect a gaze of a user within a gaze field of view;
[0077] a display having a display area positioned to overlap a portion of the gaze field, the display area being contained within the gaze field; and
[0078] a processor operably coupled to the gaze tracker and the display, the processor configured to:
[0079] detecting movement of the user's gaze to an activation area of the gaze field outside the display area; and
[0080] In response to detecting movement of the user's gaze toward the activation area, a function of the electronic device is activated.
[0081] Clause 2. The electronic device according to clause 1, further comprising:
[0082] a frame containing the gaze tracker and the display, the frame defining a frame area positioned to overlap the gaze field of view, the display area being contained within the frame area, the frame area being contained within the gaze field of view, wherein:
[0083] The activation area is outside the frame area.
[0084] Clause 3. An electronic device according to any one of clauses 1 to 2, wherein:
[0085] The processor is configured to cause a feedback signal to be provided to the user in response to detecting movement of the user's gaze to the activation area.
[0086] Clause 4. The electronic device of clause 3, wherein:
[0087] The strength of the feedback signal increases in proportion to the amount of time the user's gaze remains in the activation area.
[0088] Clause 5. An electronic device according to any one of clauses 3 to 4, wherein:
[0089] The strength of the feedback signal increases in proportion to the proximity of the user's gaze to the activation area.
[0090] Clause 6. The electronic device according to any one of clauses 3 to 5, further comprising:
[0091] a light source positioned in the frame proximate the activation area, wherein:
[0092] The feedback signal is light provided by the light source.
[0093] Clause 7. An electronic device according to any one of clauses 3 to 5, wherein:
[0094] The feedback signal is a tactile signal.
[0095] Clause 8. An electronic device according to any one of clauses 3 to 5, wherein:
[0096] The feedback signal is an audio signal.
[0097] Clause 9. An electronic device according to any one of clauses 1 to 8, wherein:
[0098] The activation areas are located in corners of the fixation field of view.
[0099] Clause 10. An electronic device according to any one of clauses 1 to 8, wherein:
[0100] The activation area is adjacent to an edge of the fixation field of view.
[0101] Clause 11. An electronic device according to any one of clauses 1 to 10, wherein:
[0102] The processor is configured to activate the function in response to detecting that a gaze of the user remains in the activation area for an amount of time greater than a threshold amount of time.
[0103] Clause 12. An electronic device, comprising:
[0104] a gaze tracker configured to detect a gaze of a user within a gaze field of view; and
[0105] a processor operably coupled to the gaze tracker, the
[0106] The processor is configured as:
[0107] detecting movement of the user's gaze toward an activation area of the gaze field;
[0108] determining one or more characteristics of movement of the user's gaze toward the activation area;
[0109] determining whether the user's gaze remains within the activation area for at least a dwell time, the dwell time being based on the one or more characteristics of the movement of the user's gaze; and
[0110] In response to determining that the user's gaze remains within the activation area for at least the dwell time, a function of the electronic device is activated.
[0111] Clause 13. An electronic device according to clause 12, wherein the movement of the user's gaze
[0112] The one or more characteristics of the user's gaze include a speed of the movement of the user's gaze. Clause 14. An electronic device according to clause 13, wherein the dwell time is related to the user's
[0113] The gaze is inversely proportional to the movement speed.
[0114] Clause 15. An electronic device according to any one of clauses 12 to 14, wherein the one or more characteristics of the movement of the user's gaze include a path shape of the movement of the user's gaze from an initial point to the activation area.
[0115] Clause 16. The electronic device of clause 15, wherein the dwell time is at least partially based on
[0116] The relationship between the path shape of the movement of the user's gaze and the boundary of the activation area.
[0117] Clause 17. An electronic device according to any one of clauses 12 to 16, wherein the electronic device further comprises:
[0118] include:
[0119] A display having a display area positioned to overlap a portion of the gaze field, the display area being contained within the gaze field, the activation area being outside the display area.
[0120] Clause 18. The electronic device according to clause 17, further comprising:
[0121] A frame includes the gaze tracker and the display, the frame defining a frame area positioned to overlap the gaze field of view, the display area being contained within the frame area, and the frame area being contained within the gaze field of view.
[0122] Clause 19. An electronic device according to any one of clauses 12 to 18, wherein:
[0123] The activation areas are located in corners of the fixation field of view.
[0124] Clause 20. An electronic device according to any one of clauses 12 to 18, wherein:
[0125] The activation area is adjacent to an edge of the fixation field of view.
[0126] Clause 21. An electronic device, comprising:
[0127] one or more sensors;
[0128] a gaze tracker configured to detect a gaze of a user within a gaze field of view; and
[0129] a processor operatively coupled to the one or more sensors
[0130] and the gaze tracker, the processor being configured to:
[0131] detecting movement of the user's gaze toward an activation area of the gaze field;
[0132] modifying the activation area based on one or more sensor signals from the one or more sensors;
[0133] In response to detecting movement of the user's gaze to the modified activation area, a function of the electronic device is activated.
[0134] Clause 22. The electronic device of clause 21, wherein:
[0135] Modifying the activation region includes enabling and disabling the activation region; and
[0136] When the activation area is disabled, the function of the electronic device is not activated. Clause 23. An electronic device according to any one of clauses 21 to 22, wherein:
[0137] Modifying the activation area includes changing a size of the activation area.
[0138] Clause 24. An electronic device according to any one of clauses 21 to 23, wherein:
[0139] The activation area is associated with a function of the electronic device; and
[0140] Modifying the activation area is based on a functionality of the electronic device associated with the activation area.
[0141] Clause 25. An electronic device according to any one of clauses 21 to 24, wherein:
[0142] The one or more sensor signals are indicative of motion of the electronic device.
[0143] Clause 26. An electronic device according to any one of clauses 21 to 25, wherein the electronic device further comprises:
[0144] include:
[0145] The one or more sensors include a camera having an imaging field of view that at least partially overlaps the gaze field of view and includes the activation area, wherein:
[0146] The one or more sensor signals indicate whether the camera detects the presence of an object matching object criteria in the activation area.
[0147] Clause 27. The electronic device of clause 26, wherein:
[0148] The object criteria include a type of the object, a size of the object, and a proximity of the object to the electronic device.
[0149] Clause 28. An electronic device according to any one of clauses 21 to 27, wherein:
[0150] The one or more sensor signals indicate that the user is engaged in an activity matching activity criteria.
[0151] Clause 29. An electronic device according to any one of clauses 21 to 28, wherein the electronic device further comprises:
[0152] include:
[0153] A display having a display area positioned to overlap a portion of the gaze field, the display area being contained within the gaze field, and the activation area being outside the display area.
[0154] Clause 30. The electronic device according to clause 29, further comprising:
[0155] A frame includes the gaze tracker and the display, the frame having a frame area positioned to overlap the gaze field of view, the display area being contained within the frame area, and the frame area being contained within the gaze field of view.
[0156] Clause 31. An electronic device according to any one of clauses 21 to 30, wherein:
[0157] The activation areas are located in corners of the fixation field of view.
[0158] Clause 32. An electronic device according to any one of clauses 21 to 31, wherein:
[0159] The activation area is adjacent to an edge of the fixation field of view.
Claims
1. An electronic device, comprising: a gaze tracker configured to detect a gaze of a user within a gaze field of view; a display having a display area positioned to overlap a portion of the gaze field, the display area being contained within the gaze field; and a processor operably coupled to the gaze tracker and the display, the processor configured to: detecting movement of the user's gaze to an activation area of the gaze field outside the display area; as well as In response to detecting movement of the user's gaze toward the activation area, a function of the electronic device is activated.
2. The electronic device according to claim 1, further comprising: a frame containing the gaze tracker and the display, the frame defining a frame area positioned to overlap the gaze field of view, the display area being contained within the frame area, the frame area being contained within the gaze field of view, wherein: The activation area is outside the frame area.
3. The electronic device according to any one of claims 1 to 2, wherein: The processor is configured to cause a feedback signal to be provided to the user in response to detecting movement of the user's gaze to the activation area.
4. The electronic device according to claim 3, wherein: The strength of the feedback signal increases in proportion to the amount of time the user's gaze remains in the activation area.
5. The electronic device according to any one of claims 3 to 4, wherein: The strength of the feedback signal increases in proportion to the proximity of the user's gaze to the activation area.
6. The electronic device according to any one of claims 3 to 5, further comprising: a light source positioned in the frame proximate the activation area, wherein: The feedback signal is light provided by the light source.
7. The electronic device according to any one of claims 3 to 5, wherein: The feedback signal is a tactile signal.
8. The electronic device according to any one of claims 3 to 5, wherein: The feedback signal is an audio signal.
9. The electronic device according to any one of claims 1 to 8, wherein: The activation areas are located in corners of the fixation field of view.
10. The electronic device according to any one of claims 1 to 8, wherein: The activation area is adjacent to an edge of the fixation field of view.
11. The electronic device according to any one of claims 1 to 10, wherein: The processor is configured to activate the function in response to detecting that a gaze of the user remains in the activation area for an amount of time greater than a threshold amount of time.
12. An electronic device, comprising: a gaze tracker configured to detect a gaze of a user within a gaze field of view; and a processor operably coupled to the gaze tracker, the processor configured to: detecting movement of the user's gaze toward an activation area of the gaze field; determining one or more characteristics of movement of the user's gaze toward the activation area; determining whether the user's gaze remains within the activation area for at least a dwell time, the dwell time being based on the one or more characteristics of the movement of the user's gaze; as well as In response to determining that the user's gaze remains within the activation area for at least the dwell time, a function of the electronic device is activated. 13 . The electronic device of claim 12 , wherein the one or more characteristics of the movement of the user's gaze include a speed of the movement of the user's gaze. The electronic device according to claim 13 , wherein the dwell time is inversely proportional to the moving speed of the user's gaze.
15. The electronic device according to any one of claims 12 to 14, wherein the one or more characteristics of the movement of the user's gaze include a path shape of the movement of the user's gaze from an initial point to the activation area.
16. The electronic device of claim 15, wherein the dwell time is based at least in part on a relationship between a path shape of the movement of the user's gaze and a boundary of the activation area.
17. The electronic device according to any one of claims 12 to 16, further comprising: A display having a display area positioned to overlap a portion of the gaze field, the display area being contained within the gaze field, the activation area being outside the display area.
18. The electronic device according to claim 17, further comprising: A frame includes the gaze tracker and the display, the frame defining a frame area positioned to overlap the gaze field of view, the display area being contained within the frame area, and the frame area being contained within the gaze field of view.
19. The electronic device according to any one of claims 12 to 18, wherein: The activation areas are located in corners of the fixation field of view.
20. The electronic device according to any one of claims 12 to 18, wherein: The activation area is adjacent to an edge of the fixation field of view.
