Gaze-based touch interaction for notifications

Abstract

Methods and apparatus for gaze-based device interaction are described. During the display of a notification on a touch-sensitive display screen of an electronic device, gaze tracking is performed to determine the location of the gaze point. Within a defined time period, the nearest gaze point location is determined to fall within a proxy area of ​​the notification. In response to determining that the nearest gaze point location falls within the proxy area of ​​the notification, the electronic device is operated in a one-handed mode. In the one-handed mode, touch input detected on the touch-sensitive display screen, outside the notification, is processed as an interaction with the notification.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Application No. 18 / 884,476, filed September 13, 2024, and U.S. Application No. 18 / 412,926, filed January 15, 2024, which are incorporated herein by reference in their entirety. Technical Field

[0002] This invention relates to methods and apparatus for implementing gaze-based device interaction, and in particular gaze-based touch interaction for notification. Background Technology

[0003] Touch-based user interaction on handheld devices (such as smartphones) typically requires the user to touch certain areas of a touch-sensitive display. When the device is held in a typical one-handed posture, the thumb of the hand holding the device is usually used for touch input. However, the reach of the thumb is usually limited, making it difficult for users to provide touch input in certain areas of the touch-sensitive display. This problem is becoming increasingly serious as touch-sensitive displays continue to grow in size.

[0004] There is a need to provide a solution that can improve user interaction with touch-sensitive displays. Summary of the Invention

[0005] In various examples, the present invention describes methods and apparatus for providing gaze-based touch interaction. Gaze detection can be used to enable a user to interact with notifications displayed on a touch-sensitive display in an area that would otherwise be outside the user's touch range (e.g., outside the reach of the thumb when the user is holding a handheld device). This provides a more efficient way of user interaction because the user can conveniently provide touch input using the same hand holding the device. Furthermore, examples of the present invention can support more intuitive user interaction for users with physical limitations.

[0006] In some examples, the present invention describes a gaze-based mechanism that enables a device to switch between a one-handed mode (sometimes called an active mode) and a default mode (sometimes called an inactive mode) based on user gaze detected in an area defined relative to the displayed notification. In one-handed mode, touch input outside the area of ​​the displayed notification can be used to interact with the notification. In the second mode, conventional interaction mechanisms (e.g., touch input that needs to be within the displayed notification to interact with the notification) can be supported. Technical advantages include using gaze detection to enable users to interact with displayed notifications that would otherwise be outside the user's touch range. For example, one-handed interaction can be enabled (the user can interact with the notification using the same hand holding the device).

[0007] Using gaze tracking to enable one-handed mode helps avoid false alarms (e.g., a user's touch input being incorrectly interpreted as interaction with a displayed notification) while simplifying the user's body movements when interacting with the notification. Examples of the invention also provide user interface (UI) designs that conveniently inform the user of the available options for interacting with the displayed notification.

[0008] In one exemplary aspect, the present invention describes a method in an electronic device, the method comprising: performing gaze tracking to determine a gaze point location while displaying a notification on a touch-sensitive display screen of the electronic device; determining, within a defined time period, that the most recent gaze point location falls within a proxy area of ​​the notification; and, in response to determining that the most recent gaze point location falls within the proxy area of ​​the notification, operating the electronic device in a one-handed mode, wherein, in the one-handed mode, touch input detected on the touch-sensitive display screen outside the notification is processed as an interaction with the notification.

[0009] In one example of the above exemplary aspect of the method, determining that the most recent gaze location falls within the proxy region of the notification may include: using a sliding window to define the most recent gaze location, the sliding window corresponding to a time window shorter than the defined time period; and determining that a threshold number of gaze locations within the sliding window fall within the proxy region of the notification.

[0010] In one example of any of the above exemplary aspects of the method, determining that the most recent gaze position falls within the proxy region of the notification may include: storing the determined gaze position at each time step in a first-in-first-out buffer, the size of which corresponds to the time window of the sliding window; and determining that a threshold number of gaze positions stored in the buffer fall within the proxy region of the notification.

[0011] In one example of any of the above exemplary aspects of the method, the proxy region may be defined as encompassing the display area of ​​the notification and extending beyond the display area of ​​the notification.

[0012] In one example of any of the above exemplary aspects of the method, the method may include, during operation of the electronic device in the one-handed mode: detecting the touch input on the touch-sensitive display screen outside the display area of ​​the notification; and processing the touch input as the interaction with the notification.

[0013] In one example of any of the above exemplary aspects of the method, the method may include, during operation of the electronic device in the one-handed mode, further comprising: detecting another touch input on the touch-sensitive display screen within the display area of ​​the notification; ending operation of the electronic device in the one-handed mode, and processing the other touch input as an interaction with the notification.

[0014] In any example of any of the above exemplary aspects of the method, the method may include, during operation of the electronic device in the one-handed mode: displaying a one-handed user interface (UI), the one-handed UI including at least one of the following: a visual indicator indicating a type of touch input for interacting with the notification; a UI element displayed in a thumb-accessible area of ​​the touch-sensitive display for interacting with the notification; a visual element indicating operation in the one-handed mode; or a visual modification of the notification.

[0015] In one example of any of the above exemplary aspects of the method, the notification may be displayed in an area of ​​the touch-sensitive display that is not accessible to the thumb.

[0016] In one example of any of the above exemplary aspects of the method, the method may include: terminating gaze tracking in response to determining that the nearest gaze location falls within the proxy area of ​​the notification.

[0017] In another exemplary aspect, the present invention describes an electronic device comprising: a processing unit configured to execute instructions to cause the electronic device to: perform gaze tracking to determine a gaze point location while displaying a notification on a touch-sensitive display screen of the electronic device; determine, within a defined time period, that the nearest gaze point location falls within a proxy area of ​​the notification; and, in response to determining that the nearest gaze point location falls within the proxy area of ​​the notification, operate the electronic device in a one-handed mode, wherein, in the one-handed mode, touch input detected on the touch-sensitive display screen outside the notification is processed as an interaction with the notification.

[0018] In one example of the above exemplary aspects of the electronic device, the processing unit may be configured to execute the instructions to cause the electronic device to determine that the most recent gaze location falls within the proxy area of ​​the notification by: defining the most recent gaze location using a sliding window, the sliding window corresponding to a time window shorter than the defined time period; and determining that a threshold number of gaze locations in the sliding window fall within the proxy area of ​​the notification.

[0019] In one example of any of the above exemplary aspects of the electronic device, the processing unit may be configured to execute the instructions to cause the electronic device to determine that the most recent gaze position falls within the proxy region of the notification by: storing the determined gaze position at each time step in a first-in-first-out buffer, the size of the buffer corresponding to the time window of the sliding window; and determining that a threshold number of gaze positions stored in the buffer fall within the proxy region of the notification.

[0020] In one example of any of the above exemplary aspects of the electronic device, the proxy area may be defined as covering the display area of ​​the notification and extending beyond the display area of ​​the notification.

[0021] In one example of any of the above exemplary aspects of the electronic device, the processing unit may be configured to execute the instructions to cause the electronic device, while operating the electronic device in the one-handed mode, to: detect the touch input on the touch-sensitive display screen outside the display area of ​​the notification; and process the touch input as the interaction with the notification.

[0022] In one example of any of the above exemplary aspects of the electronic device, the processing unit may be configured to execute the instructions to cause the electronic device, while operating the electronic device in the one-handed mode, to: detect another touch input on the touch-sensitive display screen within the display area of ​​the notification; terminate operation of the electronic device in the one-handed mode; and process the other touch input as an interaction with the notification.

[0023] In one example of any of the above exemplary aspects of the electronic device, the processing unit may be configured to execute the instructions to cause the electronic device to: display a one-handed user interface (UI) during operation of the electronic device in the one-handed mode, the one-handed UI including at least one of the following: a visual indicator indicating the type of touch input for interacting with the notification; a UI element displayed in a thumb-accessible area of ​​the touch-sensitive display for interacting with the notification; a visual element indicating operation in the one-handed mode; or a visual modification of the notification.

[0024] In one example of any of the above exemplary aspects of the electronic device, the notification may be displayed in an area of ​​the touch-sensitive display that is not accessible to the thumb.

[0025] In one example of any of the above exemplary aspects of the electronic device, the processing unit may be configured to execute the instruction to cause the electronic device to terminate gaze tracking in response to determining that the nearest gaze location falls within the proxy area of ​​the notification.

[0026] In one example of any of the above exemplary aspects of the electronic device, the electronic device may be a handheld device.

[0027] In another exemplary aspect, the present invention describes a non-transient computer-readable medium storing instructions that are executable by a processing unit of an electronic device to cause the electronic device to: perform gaze tracking to determine a gaze point location while a notification is displayed on a touch-sensitive display screen of the electronic device; determine, within a defined time period, that the nearest gaze point location falls within a proxy area of ​​the notification; and, in response to determining that the nearest gaze point location falls within the proxy area of ​​the notification, operate the electronic device in a one-handed mode, wherein, in the one-handed mode, touch input detected on the touch-sensitive display screen outside the notification is processed as an interaction with the notification.

[0028] In one example of the above exemplary aspects of the non-transitory computer-readable medium, the instructions may cause the electronic device to perform any of the above exemplary aspects of the method.

[0029] In another exemplary aspect, the present invention describes a processing module for controlling a device to cause the device to perform any of the above-described exemplary aspects of the method.

[0030] In another exemplary aspect, the present invention describes a system-on-a-chip including a processing unit for executing instructions to cause a device to perform any of the above-described exemplary aspects of the method.

[0031] In another exemplary aspect, the present invention describes a computer program characterized in that, when the computer program is run on a computer, it causes the computer to perform any of the above-described exemplary aspects of the method. Attached Figure Description

[0032] The accompanying drawings, which illustrate exemplary embodiments of this application, will now be shown by way of example, in which: Figure 1A A block diagram illustrating an example of an electronic device that can be used to implement an example of the present invention; Figure 1B A front view of an exemplary electronic device that can be used to implement examples of the present invention is shown; Figures 2A to 2C An example of gaze-based device interaction for interacting with displayed notifications, provided by an example of the present invention, is shown. Figure 3 This is a flowchart illustrating an exemplary method for gaze-based device interaction provided by an example of the present invention; Figure 4 An example of a one-handed user interface provided by the present invention is shown.

[0033] Similar reference numerals may be used in different figures to represent similar components. Detailed Implementation

[0034] In various examples, the present invention describes methods and apparatuses that enable gaze-based touch interaction for interacting with notifications (e.g., pop-up notification windows) on electronic devices (e.g., handheld devices, such as smartphones). In this invention, the term "touch input" is intended to cover various types of input that may be detected by an electronic device due to contact or proximity of an object (e.g., a user's finger or stylus) with the electronic device. For example, touch input can be input detected by capacitive sensors, force sensors, or pressure sensors, and combinations thereof. Therefore, the term "touch input" can encompass force input and pressure input.

[0035] In this invention, the electronic device can be any device with a display screen, including mobile communication devices (e.g., smartphones), tablet devices, laptop devices, desktop computer devices, vehicle-based devices (e.g., infotainment systems or interactive dashboard devices), interactive self-service terminal devices, or Internet of Things (IoT) devices, etc. In particular, this invention can cover any electronic device with a touch-sensitive display screen, including handheld devices with touch-sensitive displays.

[0036] Figure 1A This is a block diagram illustrating some components of an exemplary electronic device 100 (also generally referred to as a device) that can be used to implement embodiments of the present invention. Although exemplary embodiments of the electronic device 100 are shown and discussed below, other embodiments may be used to implement the examples disclosed herein, which may include components different from those shown. Figure 1A A single instance of each component is shown, but each component shown can have multiple instances.

[0037] Electronic device 100 includes one or more processing units 102, such as processors, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), dedicated logic circuits, graphics processing units (GPUs), central processing units (CPUs), neural processing units (NPUs), dedicated artificial intelligence processor units, or combinations thereof. Electronic device 100 also includes one or more input / output (I / O) interfaces 104 connected to one or more I / O devices 110, such as cameras 112, touch-sensitive displays 114 (also called touchscreens or simply displays), optional microphones 116, optional pressure sensors 118 (also called force sensors), optional speakers 120, and / or optional haptic units 122 (also called vibration units). Electronic device 100 may include other input devices (e.g., physical buttons, keyboards, etc.) and / or other output devices (e.g., lights, etc.).

[0038] Electronic device 100 may include one or more optional network interfaces 106 for wired or wireless communication with a network (e.g., intranet, internet, P2P network, WAN, and / or LAN) or other nodes. Network interface 106 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for intra-network and / or inter-network communication. In some examples, network interface 106 may enable electronic device 100 to communicate with a network to access cloud-based services.

[0039] Electronic device 100 includes one or more memories 108, which may include volatile or non-volatile memories (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). Non-transient memories 108 may store instructions executable by processing unit 102, for example, to perform the examples described herein. For example, memory 108 may include instructions executable by processing unit 102 to implement gaze tracker 132, which detects and tracks a user's gaze (e.g., detecting and tracking the user's gaze position on display screen 114 using an image of the user's eyes captured by camera 112). Memory 108 may also include instructions to implement user interface (UI) controller 134, which controls interaction with the UI based on gaze detection and touch input, as further discussed below. Memory 108 may also include instructions to implement one or more software applications 136 (e.g., email applications, browser applications, calendar applications, etc.). The memory 108 may also include at least one buffer 138, which can be used to store view data, as discussed further below. The memory 108 may also include other software instructions, such as those for implementing other functions of the operating system and electronic device 100.

[0040] In some examples, electronic device 100 may also include one or more electronic storage units (not shown), such as solid-state drives, hard disk drives, disk drives, and / or optical disk drives. In some examples, one or more datasets and / or modules may be provided by external memory (e.g., an external drive that communicates with electronic device 100 via wired or wireless communication) or by transient or non-transient computer-readable media. Examples of non-transient computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable storage. For example, components of electronic device 100 may communicate with each other via a bus.

[0041] Examples of the present invention enable a user to interact with electronic device 100 using gaze input (e.g., detected by gaze tracker 132) and touch input (e.g., detected by touch-sensitive display 114). In some examples, touch input may be detected by pressure sensor 118 instead of (or in addition to) the capacitive sensor of touch-sensitive display 114. In some examples, pressure sensor 118 may be integrated with touch-sensitive display 114. In some examples, pressure sensor 118 may be separate from display 114 on electronic device 100 (e.g., touch-sensitive display 114 may be positioned in front of electronic device 100 to detect touch input, and pressure sensor 118 may be positioned behind electronic device 100) to detect pressure input. Other variations of the invention are possible.

[0042] Figure 1B This is a front view of an exemplary embodiment of electronic device 100, in which electronic device 100 is a smartphone. In some embodiments, electronic device 100 may be a handheld device that can be held with one hand. In particular, Figure 1B This illustrates how a user can hold and interact with electronic device 100 using the same hand, which can be referred to as a one-handed posture. However, it should be understood that the invention is not limited to handheld devices. For example, the invention can be applied to laptop devices, tablet devices, touch-sensitive self-service terminals, etc.

[0043] The camera 112 of the electronic device 100 may be a front-facing camera 112 capable of capturing images of the user's eyes when the user views the display screen 114. The captured images may be used by the gaze tracker 132 to detect and track the gaze point 142 (which may also be referred to as gaze input). The estimated position of the gaze point 142 may be represented as xy coordinates in the reference frame of the display screen 114 (e.g., xy coordinates in screen pixels). Touch input 144 may be detected by the display screen 114 (e.g., using a capacitive sensor, resistive sensor, pressure sensor, etc.), and the position of the touch input 144 (also referred to as the touch position) may also be represented as xy coordinates in the reference frame of the display screen 114 (e.g., xy coordinates in screen pixels). The touch input 144 may be detected over an area of ​​the display screen 114 rather than at a single point, in which case the touch position may be the centroid of the touch input 144. Both the gaze point 142 and the touch input 144 may be provided as perceived inputs (e.g., represented by their respective xy coordinates) and are used by the UI controller 134 to manage interactions with the UI displayed on the display screen 114.

[0044] In the example where electronic device 100 is a handheld device, when device 100 is held by one of the user's hands (e.g., as...), Figure 1BAs shown), the thumb of the same hand can be used to provide touch input 144 on the display screen 114. However, due to the physical limitations of the human thumb, touch input 144 using the thumb is typically limited to the lower right corner of the display screen 114 (or the lower left corner if the user holds the device 100 with their left hand). On the other hand, the user's gaze may not be subject to such physical limitations, and the gaze point 142 can be easily used as input for other areas of the display screen 114, including the upper area that the user's thumb may not easily reach.

[0045] Typically, traditional UI design aims to place interactive elements (such as soft buttons, icons, text boxes, etc.) in the lower area of ​​display screen 114 (which can be called the thumb-accessible area) so that they can be easily reached by the thumb of the same hand holding device 100. Traditional UI design usually avoids placing interactive elements in the upper area of ​​display screen 114 (which can be called the thumb-out area), where the thumb may not easily reach them. However, this results in inefficient use of display screen 114 area, as a large portion of display screen 114 becomes unusable for displaying interactive elements (and therefore "wasted"). As the size of device 100 (and therefore the area of ​​display screen 114) increases, this problem of inefficient use of display screen 114 area may become more severe, as the unusable area occupies an even larger proportion of display screen 114.

[0046] Furthermore, many traditional UI designs result in notifications (e.g., notifications of incoming messages) being displayed in the upper area of ​​display 114, which is typically outside the thumb-reachable area of ​​display 114. Displaying notifications outside the thumb-reachable area of ​​display 114 avoids them interfering with the user's current interactive task (e.g., avoiding obscuring icons displayed in the thumb-reachable area). However, the result is that it is difficult for users to interact with notifications using their thumbs. For example, if a user wants to use a swipe touch gesture to close a notification, or wants to select a notification (e.g., open an app to view an incoming message related to the notification), the user may need to hold device 100 with one hand and provide touch input with the other. This can slow down user interaction and degrade the user experience. Additionally, this can be challenging for users with physical limitations, hindering the use of both hands.

[0047] Several existing solutions aim to provide gaze-based device interaction. For example, eye-tracking systems have been developed that allow users to interact with the UI by controlling the cursor's position using gaze points. However, such eye-tracking systems are typically expensive and require time-consuming calibration to achieve acceptable levels of accuracy. Furthermore, these systems can be oversensitive to unconscious eye movements, leading to numerous false alarms. In addition, some academic research has focused on UI design based on a combination of gaze tracking and motion sensors on smartphones (e.g., Kong et al., "EyeMU Interactions: Gaze + IMU Gestures on Mobile Devices"). 2021 International Multimodal Interaction Conference Proceedings (2021). The method described by Kong et al. requires users to lock their gaze for half a second and then perform a motion gesture with their smartphone (e.g., swiping the smartphone left or right) to interact with the UI. The drawbacks are that fixing their gaze for the required amount of time may feel unnatural to users (e.g., human gazes typically move in saccades), and performing motion gestures with a smartphone may not be intuitive enough and is prone to false alarms (e.g., users may simply adjust their grip on the smartphone rather than intentionally performing a motion gesture).

[0048] In this invention, examples are described that may help address at least some of the challenges described above. Examples of the invention may provide a human-machine interface (HMI) mechanism that enables a user to interact with an electronic device using gaze-based touch interaction. In particular, examples of the invention may enable a user to interact with elements of the UI (e.g., notifications) displayed in areas that are not easily accessible when the user holds and interacts with the device in a one-handed manner.

[0049] Figures 2A to 2C An exemplary embodiment of gaze-based device interaction provided by the present invention is illustrated. Although Figures 2A to 2C The illustration is presented in the context of a smartphone as an electronic device 100, but this does not constitute a limitation.

[0050] exist Figure 2AIn this context, a plurality of interactive elements 152 (e.g., selectable icons) are displayed on the touch-sensitive display screen 114, for example, when displaying a home screen or application selection page. Specifically, the interactive elements 152 may be displayed within a first area 114a of the touch-sensitive display screen 114 (which may be the lower area of ​​the display screen 114), which is easily accessible to the user's touch (e.g., using the thumb) when the user holds and interacts with the electronic device 100 using the same hand (e.g., in a one-handed posture). The electronic device 100 is initially in a default mode in which the user interacts with the interactive elements 152 using conventional touch input.

[0051] like Figure 2A As shown, notification 154 (e.g., notification related to incoming messages such as new emails) can be displayed in a second area 114b of the display screen 114 (which may be the upper area of ​​the display screen 114) that is not easily touched by the user (e.g., using the thumb) when the user holds and interacts with the electronic device 100 with the same hand (e.g., in a one-handed posture).

[0052] In an example of the invention, display notification 154 may trigger the detection and tracking of a user's gaze within a defined time period (e.g., 1 to 2 seconds from the display notification 154) (e.g., capturing an image of the user's eyes using camera 112 and tracking the gaze using gaze tracker 132 based on the captured image). Within the defined time period, gaze tracker 132 may detect the position of gaze point 142 at different time steps (e.g., one gaze point 142 is detected for each frame captured by camera 112). The position of gaze point 142 (e.g., x, y coordinates based on the reference frame of display screen 114) may be stored in buffer 138. Buffer 138 may be a first-in first-out (FIFO) buffer storing the position data of the most recent gaze point 142. Buffer 138 may be used to implement a sliding window that corresponds to a time shorter than the defined time period (e.g., the most recent 0.5 seconds). For example, if gaze tracker 132 processes frames at 20 fps, the size of buffer 138 may be set to store the last ten gaze points 142 (corresponding to the last 0.5-second sliding window). It should be noted that the size of the buffer 138, the size of the sliding window, and the length of the defined time period can all be optional parameters (e.g., depending on the frame rate of the camera 112 and / or the speed of the gaze tracker 132). Similarly, other techniques can also be used to implement the sliding window.

[0053] The UI controller 134 (or other system-level controller) can check whether the position of the most recently viewed point 142 in the sliding window falls within a proxy region 156, which is defined as at least covering the display area of ​​the notification 154. Specifically, the proxy region 156 can extend to cover an area larger than the displayed notification 154. For example, the proxy region 156 can extend 1 cm beyond the left, right, top, and bottom boundaries of the displayed notification 154. It should be noted that the proxy region 156 can extend to different extents in different directions, and the proxy region 156 can be of any shape or size. Figure 2A As shown, in some examples, the proxy region 156 may extend beyond the display 114 and / or beyond the device 100. The proxy region 156 may be defined as a display area larger than the notification 154 to allow for inaccuracies and / or imprecisions that may occur at the determined gaze point 142. This allows for the use of gaze trackers 132 that require less or no calibration, less processing power, and / or low-cost hardware. Furthermore, using the proxy region 156 allows the notification 154 to be identified as the focus of the user's gaze without requiring fixed gaze (e.g., allowing the user's gaze to swirl within the proxy region 156), which may be more natural for the user.

[0054] If, at the end of the defined time period, the UI controller 134 (or other system-level controller) determines that no sliding window causes the position of the most recent gaze point 142 to fall within the proxy region 156 (or does not meet the threshold percentage requirement for the most recent gaze point 142 to fall within the proxy region 156), gaze tracking can automatically stop, and the electronic device 100 can maintain its default mode. Traditional touch interactions continue to be supported, and notifications 154 can continue to be displayed or can be stopped.

[0055] Optionally, during a defined time period, visual modifications to notification 154 can be provided to indicate to the user when it is determined that gaze point 142 falls within proxy region 156. For example, notification 154 can be highlighted or given a yellow border whenever gaze point 142 is determined to fall within proxy region 156. This can allow the user to shift their gaze away from proxy region 156, for example, if they are not interested in interacting with notification 154.

[0056] If, before the end of the defined time period, the UI controller 134 (or other system-level controller) determines that the positions of at least one nearest gaze point 142 in a sliding window all fall within the proxy area 156 (or a threshold percentage of the nearest gaze point 142 in the sliding window falling within the proxy area 156), then the electronic device 100 will switch to one-handed mode, and notification 154 will be identified as the target of touch interaction, such as... Figure 2BAs shown. Therefore, the detection of gaze point 142 within agent region 156 can be interpreted as the user wishing to interact with notification 154. In one-handed mode, notification 154 can be displayed with visual modifications (e.g., highlighting or adding a green border, such as...). Figure 2B As shown), to indicate that notification 154 has been identified as the target of touch interaction. The user can then interact with notification 154 using a single-handed gesture from anywhere on display 114 (including areas of display 114 outside the area where notification 154 is displayed). Specifically, as... Figure 2B As shown, touch gestures such as swiping down 146 can be performed within an area reachable by the user's thumb, which is outside the area of ​​the displayed notification 154.

[0057] When the electronic device 100 is in one-handed mode (also known as active mode) and the notification 154 is identified as the target of touch interaction, the downward swipe gesture 146 is processed as an interaction with the notification 154. For example, as Figure 2C As shown, the downward swipe gesture 146 can be processed as an interaction to open the message 156 associated with notification 154. Depending on the specific implementation, other types of touch input and interaction may also exist. In some examples, one-handed mode allows any touch interaction that would normally need to be performed within the area of ​​notification 154 (e.g., swiping down to open, swiping up to close, swiping left / right to view the next message, etc.) to be performed outside the area of ​​notification 154. Electronic device 100 can automatically switch back to default mode (or inactive mode) after the user has interacted with notification 154, allowing the user to continue interacting with electronic device 100 in the traditional way.

[0058] In some examples, electronic device 100 has switched to one-handed mode and notification 154 has been identified as the target of touch interaction (e.g., as...). Figure 2B After that (as shown), gaze tracking can be stopped. This reduces computational resource consumption because gaze tracking may no longer be necessary.

[0059] In some examples, electronic device 100 has switched to one-handed mode and notification 154 has been identified as the target of touch interaction (e.g., as...). Figure 2B Following the display of notification 154, conventional touch interactions (e.g., touch input within the area of ​​notification 154) can continue to be supported. In some examples, if the user interacts with notification 154 in a conventional manner at any time it is displayed (e.g., providing touch input within the area of ​​the displayed notification 154), electronic device 100 can automatically stop gaze tracking and switch to (or maintain) the default mode. This allows the user to continue interacting with electronic device 100 in a conventional manner, thus avoiding user frustration before they have mastered how to interact with electronic device 100 in one-handed mode.

[0060] Figure 3 This is a flowchart illustrating an exemplary method 700 for gaze-based device interaction provided by an example of the present invention. Method 700 can be provided by an electronic device (e.g., Figure 1A The electronic device 100 is implemented as follows. For example, the processing unit 102 may execute instructions in the memory 108 (e.g., instructions for the gaze tracker 132 and / or the UI controller 134) to cause the electronic device 100 to perform the exemplary method 300.

[0061] At the start of method 300, the device can operate in default mode. In default mode, traditional device interactions are supported. For example, in default mode, touch input can be used in the display area of ​​a UI element (e.g., an icon) to interact with that UI element.

[0062] At point 302, a notification is displayed on the device's touch-sensitive display. For example, an application running on the device (e.g., a messaging app) may generate and display a notification related to an incoming message. The notification may be displayed in an area of ​​the display that is not easily accessible to touch input, such as when a user holds and interacts with the handheld device using the same hand (e.g., using a one-handed gesture), where the notification is outside the reach of the user's thumb.

[0063] At 304, gaze tracking is performed (e.g., using gaze tracker 132). Gaze tracking can be performed using frames captured by the device's front-facing camera (e.g., camera 112). In some examples, gaze tracking may be activated in response to the display of a notification. In other examples, gaze tracking may have been activated before the notification is displayed (e.g., gaze tracking may always be active on the device). Gaze tracking is used to determine the location of the gaze point at each time step (where each time step may correspond to a frame captured in real time by the device's front-facing camera) (e.g., represented using x, y coordinates in the display's reference frame). Various gaze tracking techniques can be used. For example, some exemplary suitable gaze tracking techniques are described in U.S. Patent No. 11,630,510, entitled "System, method and storage medium for 2D on-screen user gaze estimation," the entire contents of which are incorporated herein by reference.

[0064] At point 306, the device determines whether the nearest gaze position falls within the notification's proxy area within a defined time period (which can be defined from the time the notification is displayed). The proxy area is defined as the area encompassing the notification's display area and potentially extending beyond it. For example, a sliding window can be used to define a set of gaze positions as the nearest gaze positions. A threshold number of gaze positions within the sliding window can be determined to fall within the notification's proxy area. The size and shape of the proxy area can be defined based on the accuracy and / or precision of the gaze positions determined by gaze tracking. For example, if the gaze tracker 132 is known to have an error of + / - 0.5 cm in all directions, the proxy area can be defined as extending 0.5 cm beyond the boundary of the notification's display area. The defined time period can be empirically defined based on typical user behavior (e.g., based on experience that a user is most likely to interact with the displayed notification within the first 2 seconds after it is displayed, the defined time period could be 2 seconds). The proxy area can be defined using x, y coordinates in the display's reference frame (e.g., if the proxy area is a rectangular area, the x, y coordinates of each corner can be used to define the proxy area), in the same reference frame as the gaze position. A sliding window captures a defined number of the nearest gaze points. Various techniques can be used to implement a sliding window. For example, optional steps 308 and 310 can be performed, where a buffer is used to implement the sliding window.

[0065] In 308, a buffer (e.g., buffer 138) is used to store the location of the most recent gaze point. The buffer can be a FIFO buffer for storing... N The nearest fixation point location (e.g., corresponding to from) N The gaze location determined by the most recently captured frame. N The value can be chosen to correspond to the desired duration of the sliding window (e.g., 0.5 seconds) and can depend on the inference speed of gaze tracking (e.g., depending on the camera's frame rate and / or the speed of the gaze tracker). For example, if the desired sliding window corresponds to 0.5 seconds and the camera's frame rate is 20 fps, then N The value can be set to 10. In some examples, the gaze positions stored in the buffer can be temporally smoothed. Various techniques can be used to perform this temporal smoothing, including but not limited to techniques using simple linear filters, Kalman filters, 1-euro filters, or other suitable temporal filtering methods.

[0066] At point 310, before the defined time period expires, determine whether the gaze location of the threshold quantity (which can be expressed as threshold quantity or threshold percentage) in the buffer falls within the proxy region of the notification. For example, if the buffer stores at least N The device can determine the location of each fixation point.N Does each storage location contain at least one ceil? pN ) fall within the defined proxy region, where, p Represents the threshold percentage ( p (A value greater than zero and less than or equal to 1) ceil() rounds up to the nearest integer equal to or greater than 1. pN The smallest integer is rounded up. The threshold percentage can be 100% (i.e., ...). p =1.0) or 80% (i.e., p =0.8), etc., and can be set based on experience.

[0067] Method 300 may remain at step 306 until the defined time period has expired or at least one sliding window occurs in which the threshold number of fixation points fall within the surrogate region, whichever occurs first. If at least one sliding window occurs within the defined time period in which the threshold number of fixation points fall within the surrogate region, then method 300 proceeds to 312.

[0068] At step 312, the device operates in one-handed mode, where the notification is identified as the interaction target. In one-handed mode, touch input outside the display area of ​​the interaction target (e.g., the notification) can be used to interact with the interaction target. Step 312 may include optional steps 314, 316, 318, 320, and 322.

[0069] At 314, optionally, gaze tracking can be terminated (e.g., stop using the front-facing camera 112 to capture images and stop executing the gaze tracker 132). This can be to reduce the consumption of computing resources (e.g., battery power, processing power, memory, etc.).

[0070] At 316, optionally, the notification can be designated as a specific interaction target. For example, UI controller 134 can make the notification highlighted or displayed with a border to indicate that the notification is an interaction target.

[0071] Optionally, at 318, a one-handed UI may be displayed. The one-handed UI may provide visual indicators to inform the user how to interact with the notification using touch input, and / or may provide UI elements for interacting with the notification within a thumb-accessible area of ​​the display. Examples of a one-handed UI are further described below.

[0072] At point 320, touch input is detected on the touch-sensitive display outside the display area of ​​the notification. For example, when the notification is displayed in an area of ​​the display that is not accessible to the thumb (e.g., the upper part of the display), touch input (e.g., a swipe gesture) can be detected in an area of ​​the display that is accessible to the thumb (e.g., the lower part of the display).

[0073] At 322, touch input is treated as an interaction with the notification (e.g., handled by UI controller 134 or other system-level controllers). For example, the type of touch input that can be treated as an interaction with the notification can be the same as conventional touch input that users use to interact with notifications (e.g., conventional swipe up, swipe down, swipe left / right gestures), except that the touch input no longer needs to be performed in the display area of ​​the notification.

[0074] After the interaction with the notification has been processed, method 300 can proceed to step 322, in which the device returns to operation in the default mode. In some examples, method 300 can proceed to step 322 if touch input has been detected in the display area of ​​the notification (e.g., the user has interacted with the notification in a conventional manner). In some examples, method 300 can automatically proceed to step 322 if no interaction with the notification has occurred after a defined time period (e.g., 5 seconds) of operation in one-handed mode.

[0075] Returning to step 306, if no at least a threshold number of gaze points fall within the sliding window of the notification's proxy area after the defined time period, method 300 can proceed to step 322, in which the device continues to operate in default mode. In some examples, if the user interacts with the displayed notification using conventional touch input during the defined time period (e.g., touch input is detected in the notification's display area), method 300 can automatically proceed from step 306 to step 322 without waiting for the defined time period to expire.

[0076] At step 322, the device operates in default mode, which supports conventional device interaction. Gaze tracking can be terminated (e.g., stopping image capture using the front-facing camera 112 and stopping gaze tracker 132). This can be to reduce the consumption of computing resources (e.g., battery power, processing power, memory, etc.). Optionally, the notification can be automatically turned off if there is no interaction with it (whether conventional or in one-handed mode).

[0077] During method 300, traditional device interactions (e.g., touch interactions in the display area of ​​a notification) can continue to be supported, and the detection of traditional interactions with the notification can cover interactions in one-handed mode. For example, if touch input is detected in the display area of ​​a notification at any time during method 300, method 300 can automatically proceed to step 322 to operate the device in default mode.

[0078] Now for reference Figure 4This illustrates an exemplary one-handed UI that can be displayed when the electronic device 100 is operating in one-handed mode (e.g., in optional step 318).

[0079] like Figure 4 As shown, when the electronic device 100 is in one-handed mode and the notification 154 has been identified as the interaction target, the notification 154 can be displayed with visual modifications (e.g., highlighting, different colored borders, etc.) to indicate that the notification 154 is the interaction target. Optionally, visual elements such as message 158 or icons can be displayed to indicate that the device 100 is operating in one-handed mode.

[0080] Optionally, one or more visual indicators 160 may be provided to indicate to the user the type of touch input they can use to interact with the notification 154 (e.g., swiping up to close, swiping down to expand). Optionally, one or more UI elements 162 may be displayed within a first area 114a of the touch-sensitive display 114 (which may be the lower area of ​​the display 114), which is easily accessible to the user's touch (e.g., using a thumb) and can be selected via touch input to interact with the notification 154. Figure 4 In the UI element 162, there are a "checkmark" element, which can be selected (by touch input on this element) to open the message related to notification 154, and an "x" element, which can be selected (by touch input on this element) to close notification 154. Other UI elements 162 can be displayed for other types of interactions, such as replying to message sending, viewing verification codes related to notification 154, etc.

[0081] It should be understood that, despite Figure 4 This demonstrates several aspects of a one-handed UI, but a one-handed UI may or may not include any combination of one or more of the visual modifications to notification 154, message 158, visual indicator 160, and / or UI element 162 described above. Typically, a one-handed UI may use only a subset of the above aspects to avoid visual clutter. In some examples, a one-handed UI can be customized by the user. For example, the user can select the position of UI element 162 to accommodate the placement of the user's thumb.

[0082] In some examples, when one or more UI elements 162 are displayed in the one-handed UI, touch input outside of UI elements 162 (e.g., in the display area of ​​notification 154, on any of the icons 154, or elsewhere on display 114) can cause the one-handed UI to automatically close, and the device 100 can automatically return to operation in the default mode.

[0083] In some examples, users can choose to display the one-handed UI when the device 100 is in one-handed mode, or not display the one-handed UI when the device 100 is in one-handed mode.

[0084] In various examples, the present invention describes methods and apparatuses for providing a gaze-based HMI mechanism that enables users to interact with notifications (or other UI elements) displayed outside of areas easily accessible to the user's touch. For example, examples of the invention can be used to enable touch interaction via the user's thumb when the user holds and interacts with a handheld device using the same hand. In various examples, a gaze-based mechanism is disclosed that allows device operation to be switched to a one-handed mode, in which touch input outside the display area of ​​a UI element that is the interaction target (e.g., a displayed notification) can be used to interact with the target.

[0085] Examples of this invention provide a mechanism (e.g., using a proxy region and a sliding window) to detect anticipated interactive targets based on the user's gaze, without requiring high accuracy or precision in gaze tracking. This offers the technical advantage of eliminating the need to calibrate the gaze tracking system and reducing the use of computational resources.

[0086] Examples of this invention may provide a more intuitive and / or convenient mechanism for users to interact with displayed notifications using their thumb when holding a handheld device in a one-handed posture. In some examples, a one-handed UI may be provided to help guide user interaction.

[0087] Although exemplary implementations using various modules and systems have been described in this invention, it should be understood that the examples of this invention can be implemented in various ways, such as as part of an operating system, as an internal function of a device, or as an application programming interface (API) function in a software development kit (SDK) that enables software developers to develop HMI mechanisms, etc.

[0088] Examples of this invention can be applied to any electronic device capable of receiving touch input (e.g., having a touch-sensitive display or other sensors capable of detecting touch) and / or communicating with a touch-sensitive input device. The electronic device may also be capable of gaze tracking and / or communicating with a gaze tracking system. Electronic devices may include smartphones, touch displays, touch-based self-service terminals, laptops, desktop computers (which may be coupled to touch-sensitive input devices), tablets, e-readers or e-ink display devices, smartwatches, and gaming devices, etc.

[0089] It should be understood that examples of the present invention may be embodied in methods, apparatus, non-transitory computer-readable media, processing modules, chipsets, system-on-a-chip, or computer programs, etc. Apparatus may include a transmitting module for performing the transmitting steps described above and a receiving module for performing the receiving steps described above. Apparatus may include processing modules, processors, or processing units for controlling or causing the apparatus to perform the examples disclosed herein.

[0090] Although the present invention describes methods and processes by steps performed in a certain order, one or more steps in the methods and processes may be omitted or modified as appropriate. One or more steps may be performed in an order other than that described, depending on the circumstances.

[0091] Although the invention has been described at least partially in terms of method, those skilled in the art will understand that the invention is also directed to various components, whether by hardware components, software, or any combination thereof, for performing at least some aspects and features of the described methods. Accordingly, the technical solutions of the invention can be embodied in the form of a software product. Suitable software products can be stored in pre-recorded storage devices or other similar non-volatile or non-transitory computer-readable media, including DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes instructions tangibly stored therein that enable a processing device (e.g., a personal computer, server, or network device) to perform examples of the methods disclosed herein. Machine-executable instructions can be in the form of sequences of code, configuration information, or other data that, when executed, cause a machine (e.g., a processor or other processing device) to perform the steps in the methods provided in the examples of the invention.

[0092] The invention may be implemented in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative in all respects and not restrictive. Features selected from one or more of the foregoing embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations will be understood within the scope of the invention.

[0093] All values ​​and subranges within the scope of disclosure are also disclosed. Furthermore, although the systems, devices, and processes disclosed and shown herein may include a specific number of elements / components, modifications may be made to say the systems, devices, and components to include more or fewer of such elements / components. For example, although any element / component disclosed may be referenced in the singular, embodiments disclosed herein may be modified to include multiple such elements / components. The subject matter described herein is intended to cover and include all suitable technical modifications.

Claims

1. A method in an electronic device, characterized in that, The method includes: During the display of a notification on the touch-sensitive display screen of the electronic device, gaze tracking is performed to determine the gaze point location; Within a defined time period, determine the location of the nearest gaze point that falls within the proxy area of ​​the notification; In response to determining that the nearest gaze point location falls within the proxy area of ​​the notification, the electronic device is operated in a one-handed mode, wherein, in the one-handed mode, touch input detected on the touch-sensitive display outside the notification is processed as an interaction with the notification.

2. The method according to claim 1, characterized in that, Determining that the nearest gaze location falls within the agent area of ​​the notification includes: A sliding window is used to define the location of the nearest fixation point, the sliding window corresponding to a time window shorter than the defined time period; The location of the gaze point in the sliding window is determined to fall within the proxy area of ​​the notification, representing a threshold number of gaze points.

3. The method according to claim 2, characterized in that, Determining that the nearest gaze location falls within the agent area of ​​the notification includes: The determined gaze point position at each time step is stored in a first-in-first-out buffer, the size of which corresponds to the time window of the sliding window; The location of the gaze point, which is determined to be a threshold number stored in the buffer, falls within the agent region of the notification.

4. The method according to any one of claims 1 to 3, characterized in that, The proxy region is defined as encompassing the display area of ​​the notification and extending beyond the display area of ​​the notification.

5. The method according to any one of claims 1 to 4, characterized in that, During operation of the electronic device in the one-handed mode, the following is also included: Detect the touch input on the touch-sensitive display screen outside the display area of ​​the notification; The touch input is processed as an interaction with the notification.

6. The method according to any one of claims 1 to 5, characterized in that, During operation of the electronic device in the one-handed mode, the following is also included: Detect another touch input on the touch-sensitive display screen within the display area of ​​the notification; The operation of the electronic device in the one-handed mode is terminated, and the other touch input is processed as the interaction with the notification.

7. The method according to any one of claims 1 to 6, characterized in that, During operation of the electronic device in the one-handed mode, the following is also included: Displaying a one-handed user interface (UI), the one-handed UI including at least one of the following: A visual indicator to indicate the type of touch input used to interact with the notification; UI elements are displayed within the thumb-accessible area of ​​the touch-sensitive display screen for interacting with the notification; Visual elements indicating operation in the one-handed mode; or Visual modifications to the notification.

8. The method according to any one of claims 1 to 7, characterized in that, The notification is displayed in an area of ​​the touch-sensitive display that is out of reach of the thumb.

9. The method according to any one of claims 1 to 8, characterized in that, Also includes: In response to determining that the location of the nearest gaze point falls within the agent area of ​​the notification, gaze tracking is terminated.

10. An electronic device, characterized in that, include: Processing unit, configured to execute instructions to cause the electronic device to: During the display of a notification on the touch-sensitive display screen of the electronic device, gaze tracking is performed to determine the gaze point location; Within a defined time period, determine the location of the nearest gaze point that falls within the proxy area of ​​the notification; In response to determining that the nearest gaze point location falls within the proxy area of ​​the notification, the electronic device is operated in a one-handed mode, wherein, in the one-handed mode, touch input detected on the touch-sensitive display outside the notification is processed as an interaction with the notification.

11. The electronic device according to claim 10, characterized in that, The processing unit is configured to execute the instructions to cause the electronic device to determine that the nearest gaze point location falls within the proxy area of ​​the notification in the following manner: A sliding window is used to define the location of the nearest fixation point, the sliding window corresponding to a time window shorter than the defined time period; The location of the gaze point in the sliding window is determined to fall within the proxy area of ​​the notification, representing a threshold number of gaze points.

12. The electronic device according to claim 11, characterized in that, The processing unit is configured to execute the instructions to cause the electronic device to determine that the nearest gaze point location falls within the proxy area of ​​the notification in the following manner: The determined gaze point position at each time step is stored in a first-in-first-out buffer, the size of which corresponds to the time window of the sliding window; The location of the gaze point, which is determined to be a threshold number stored in the buffer, falls within the agent region of the notification.

13. The electronic device according to any one of claims 10 to 12, characterized in that, The proxy region is defined as encompassing the display area of ​​the notification and extending beyond the display area of ​​the notification.

14. The electronic device according to any one of claims 10 to 13, characterized in that, The processing unit is configured to execute the instructions to cause the electronic device to operate in the one-handed mode: Detect the touch input on the touch-sensitive display screen outside the display area of ​​the notification; The touch input is processed as an interaction with the notification.

15. The electronic device according to any one of claims 10 to 14, characterized in that, The processing unit is configured to execute the instructions to cause the electronic device to operate in the one-handed mode: Detect another touch input on the touch-sensitive display screen within the display area of ​​the notification; The operation of the electronic device in the one-handed mode is terminated, and the other touch input is processed as the interaction with the notification.

16. The electronic device according to any one of claims 10 to 15, characterized in that, The processing unit is configured to execute the instructions to cause the electronic device to operate in the one-handed mode: Displaying a one-handed user interface (UI), the one-handed UI including at least one of the following: A visual indicator to indicate the type of touch input used to interact with the notification; UI elements are displayed within the thumb-accessible area of ​​the touch-sensitive display screen for interacting with the notification; Visual elements that indicate the operation in the one-handed mode; or Visual modifications to the notification.

17. The electronic device according to any one of claims 10 to 16, characterized in that, The notification is displayed in an area of ​​the touch-sensitive display that is out of reach of the thumb.

18. The electronic device according to any one of claims 10 to 17, characterized in that, The processing unit is used to execute the instructions to make the electronic device: In response to determining that the location of the nearest gaze point falls within the agent area of ​​the notification, gaze tracking is terminated.

19. The electronic device according to any one of claims 10 to 18, characterized in that, The electronic device is a handheld device.

20. A non-transitory computer-readable medium, characterized in that, It stores instructions that can be executed by a processing unit of an electronic device to cause the electronic device to perform the method according to any one of claims 1 to 9.

21. A processing module, characterized in that, For controlling the device to cause the device to perform the method according to any one of claims 1 to 9.

22. A system-on-a-chip (SoC), characterized in that, It includes a processing unit for executing instructions to cause the apparatus to perform the method according to any one of claims 1 to 9.

23. A computer program, characterized in that, When the computer program is run on a computer, it causes the computer to perform the method according to any one of claims 1 to 9.

Images