User display interface operation in scale-down mode
By displaying larger icons outside the scaled-down area and using gaze detection, the challenges of obscured content and difficult interaction in one-handed smartphone use are addressed, enhancing readability and usability.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TELEFONAKTIEBOLAGET LM ERICSSON (PUBL)
- Filing Date
- 2021-11-03
- Publication Date
- 2026-05-15
AI Technical Summary
Existing one-handed operation techniques for touch-based user interfaces on large smartphones face challenges such as difficulty in reading and writing smaller content, and input elements being obscured by the user's finger, especially when scaling down the display.
Utilizing screen space outside the scaled-down display area to display larger images of application icons or information items, and employing user gaze detection to interact with these items without requiring finger reach, along with gesture-based interactions to facilitate one-handed operation.
Enables easier readability and interaction with important information by displaying larger icons and using gaze detection, simplifying one-handed operation on mobile devices.
Smart Images
Figure 0007860229000001 
Figure 0007860229000002 
Figure 0007860229000003
Abstract
Description
Technical Field
[0001] The present invention relates to a technology that enables a user to operate a handheld device having a touch screen display, and more particularly, to a technology that enables a user to use one hand to hold the device while at the same time having full access to a scaled-down interface presented via the touch screen display.
[0002] Today's smartphones not only display information to the user, but also have a touch screen that enables the user to supply inputs such as selections and data to the device. The user mainly interacts with the touch screen by touching the display with one or more fingers. However, generally, touch interaction can have many types of gesture inputs (micro-interactions) such as taps, double-taps, slides, tap-slides, tap-holds, swipes, flips, pinches, etc.
[0003] Entering information in this way is extremely easy and accurate when the user holds the device with one hand and interacts with the touch screen with the other hand. However, in a very large number of cases, the user is busy doing other things such as holding a bag with the other hand while holding the device with one hand. Relatively long ago, when telephones were small and had physical buttons only on a part of the front, for most people, using only one hand for both operating the telephone and holding the telephone (i.e., single-handed operation) was relatively easy. However, in the case of today's large telephones, this is extremely difficult with a touch-based user interface (UI), and as a result, it is very common for people to drop the telephone while trying to do so.
[0004] To address this issue, some current devices feature technology that allows users to operate the phone with one hand by temporarily scaling down the screen display size, making it easier to control the phone with only one hand. The scaled display can be moved around on the screen by touching the scaled display window bar and moving the window. The scaled display can be made smaller or larger by dragging one of the corners of the scaled display window.
[0005] Other solutions are also known. For example, J. Lai et al., "ThumbStroke: A Virtual Keyboard in Support of Sight-Free and One-Handed Text Entry on Touchscreen Mobile Devices," ACM Trans.Manage.Inf.Syst., Vol.10, No.3, Article 11, published in September 2019, describes a virtual keyboard that enables one-handed text entry on mobile devices. One possible way to activate the keyboard is by using the hot corners on the device.
[0006] U.S. Patent Publication No. 201900731258, published on March 7, 2019, describes an alternative solution for keyboards to adapt to the characteristics of the hand and thumb.
[0007] The publication corresponding to international application number PCT / EP2021 / 064294 describes a technique that allows a user to efficiently input gestures that define both the size and position of a scaled-down display window on a screen while simultaneously holding a user device in one hand.
[0008] Known techniques that utilize scaling address some of the problems associated with one-handed operation of a device, but related issues and / or drawbacks still remain. For example, none of the solutions mentioned above address the related problem that scaling down the display window on the screen makes it difficult to read and write content, the entire user interface and input elements such as buttons become smaller, and therefore the content is often obscured (and thus invisible) by the finger being used to operate it (most often the thumb in one-handed mode).
[0009] Therefore, there is a need for technology to address the above-mentioned problems and / or related issues. [Overview of the project]
[0010] The terms “comprises” and “comprising,” as used herein, are used to indicate the presence of the described features, integers, steps, or components, but it should be emphasized that the use of these terms does not preclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.
[0011] Furthermore, reference letters may be given in some cases (for example, in the claims and abstract) to facilitate the identification of various steps and / or elements. However, the use of reference letters is not intended to imply or suggest that the steps and / or elements thus referenced should be performed or operated in any particular order.
[0012] According to one aspect of the present invention, the aforementioned and other objectives are achieved in a technique (e.g., method, apparatus, non-temporary computer-readable storage medium, program means) for operating a user interface of a device, the user interface including a touch-sensitive display. The operation includes detecting that a mode change gesture has been received by the touch-sensitive display while operating the device in full-size mode. In response to the detection, the user interface is operated in scale mode, and when an application or system software generates an output image to be displayed on the device, Operating the user interface in full-size mode is Displaying the output image on the first area of the touch-sensitive display, This includes accepting touch input from an application or system software in a set of input touch points located within a first area of a touch-sensitive display, Operating the user interface in scale mode is The process involves scaling an output image and displaying the scaled output image on a scaled area of a touch-sensitive display, wherein the scaled area is smaller than the first area. Scaling a set of input touchpoints so that the scaled set of input touchpoints lies within the scaled area of the touch-sensitive display, Displaying one or more information items, including a first information item, within the remaining interface area of a touch-sensitive display, wherein the remaining interface area is part of the touch-sensitive display located outside the scaled area of the touch-sensitive display, and the information items are displayed within that area. Obtaining user image data, This includes, through analysis of image data, detecting when the user is fixating on a first information item within the remaining interface area, and in response to that detection, displaying a highlighted image of that first information item and / or activating the application corresponding to the first information item.
[0013] In some, though not all, embodiments consistent with the present invention, operating the user interface in scale mode further includes displaying an image generated by an application within a scaled area of a touch-sensitive display when the application corresponding to a first information item is activated.
[0014] In some, though not all, embodiments consistent with the present invention, the first information item is one of several information items displayed within the remaining interface area of the touch-sensitive display when the user interface is operated in scale mode, and the operation of the device's user interface includes, from the analysis of image data, detecting which of the several information items the user is fixating on, displaying a highlighted image of the detected information item, and / or activating an application corresponding to the detected information item.
[0015] In some, but not all, embodiments consistent with the present invention, the operation of the device's user interface includes detecting that a scaled application icon currently displayed in a scaled area of the touch-sensitive display is covered by the user's finger on the touch-sensitive display, and in response to detecting that a scaled application icon is covered by the user's finger on the touch-sensitive display, displaying an unscaled application icon as one of one or more information items displayed in the remaining interface area of the touch-sensitive display, the unscaled application icon corresponding to the scaled application icon.
[0016] In some, though not all, embodiments consistent with the present invention, user interface operation includes detecting that an area of a scaled area of the touch-sensitive display is covered by the user's finger on the touch-sensitive display, and in response to detecting that an area of a scaled area of the touch-sensitive display is covered by the user's finger on the touch-sensitive display, displaying an unscaled rendering of the scaled image displayed in the area of the scaled area of the touch-sensitive display as one of one or more information items displayed in the remaining interface area of the touch-sensitive display.
[0017] In some, though not all, embodiments consistent with the present invention, operating the user interface in scale mode includes detecting which of a plurality of applications was most recently activated when the user interface was operated in full-size mode, selecting an application icon corresponding to the detected application among the plurality of applications, and displaying the selected application icon as one or more information items in the remaining interface area of the touch-sensitive display.
[0018] In yet another aspect of some embodiments consistent with the present invention, though not necessarily all, operating the user interface in scale mode includes detecting which of several applications was the most frequently activated application when the user interface was operated in full-size mode, selecting an application icon corresponding to the detected application among the several applications that was the most frequently activated application, and displaying the selected application icon as one or more information items in the remaining interface area of the touch-sensitive display.
[0019] In some, though not all, such embodiments consistent with the present invention, detecting which of several applications was the most frequently activated application when the user interface is operated in full-size mode includes detecting which of several applications was the most frequently activated application during a predetermined number of recent periods of time when the user interface is operated in full-size mode.
[0020] In some, though not all, embodiments consistent with the present invention, at least one of one or more information items is a text notification received and / or generated by the device.
[0021] In some, though not all, embodiments consistent with the present invention, the operation of the user interface includes detecting that the user gazes at one or more information items in the remaining interface area for a period of time less than or equal to a predetermined amount of time, and then shakes the device, and in response to the detection, displaying a highlighted image of one or more information items.
[0022] In some such embodiments consistent with the present invention, though not necessarily all, the operation of the user interface includes detecting that the user tilts the device and, in response, replacing a highlighted image of one or more information items with a non-highlighted image of one or more information items and displaying a different highlighted image of one or more information items.
[0023] In some, but not all, embodiments, the techniques described in any one or various combinations of the above embodiments are embodied in a computer program that includes instructions causing at least one processor to perform the operations of the user interface described above, when executed by at least one processor. In some, but not all, such embodiments, the techniques take the form of a carrier including a computer program, the carrier being one of electronic signals, optical signals, radio signals, and computer-readable storage media.
[0024] In some, though not all, embodiments of this technology are embodied as a device.
[0025] The objects and advantages of the present invention will be understood by reading the following detailed description in conjunction with the drawings.
Brief Description of the Drawings
[0026] [Figure 1A] A diagram showing a device having a touch-sensitive display that is initially operated in full-size mode and transitions to a scaled operation mode as a result of a user applying a gesture to the touch-sensitive display. [Figure 1B] A diagram showing a device having a touch-sensitive display that is initially operated in full-size mode and transitions to a scaled operation mode as a result of a user applying a gesture to the touch-sensitive display. [Figure 1C] A diagram showing a device having a touch-sensitive display that is initially operated in full-size mode and transitions to a scaled operation mode as a result of a user applying a gesture to the touch-sensitive display. [Figure 1D] A diagram showing a device having a touch-sensitive display that is initially operated in full-size mode and transitions to a scaled operation mode as a result of a user applying a gesture to the touch-sensitive display. [Figure 1E] A diagram showing a device having a touch-sensitive display that is initially operated in full-size mode and transitions to a scaled operation mode as a result of a user applying a gesture to the touch-sensitive display. [Figure 1F] A diagram showing a device having a touch-sensitive display that is initially operated in full-size mode and transitions to a scaled operation mode as a result of a user applying a gesture to the touch-sensitive display. [Figure 2] A diagram of a scaled screen operation mode of a device according to one aspect of an embodiment of the present invention. [Figure 3] A flowchart of actions performed by a device configured for single-handed operation according to one aspect of an embodiment of the present invention. [Figure 4] This is a block diagram of an exemplary controller for a device, with some exemplary embodiments consistent with the present invention, though not necessarily all. [Modes for carrying out the invention]
[0027] Next, various features of the present invention will be described in relation to several exemplary embodiments, with reference to figures in which similar parts are identified by the same reference numerals.
[0028] To facilitate understanding of the present invention, many aspects of the invention will be described in relation to sequences of actions performed by computer systems or other hardware elements capable of executing programmed instructions. In each embodiment, it will be recognized that various actions can be performed by dedicated circuits (e.g., analog and / or discrete-element logic gates interconnected to perform dedicated functions), by one or more processors programmed with appropriate sets of instructions, or a combination of both. The term “circuit configured to perform one or more of the described actions” is used herein to refer to any such embodiment (i.e., one or more dedicated circuits only, one or more programmed processors, or any combination thereof). Furthermore, it can be further considered that the present invention is fully embodied in any form of non-transient computer-readable carrier, such as solid-state memory, magnetic disk, or optical disk, containing an appropriate set of computer instructions that cause a processor to perform the techniques described herein. Thus, various aspects of the present invention may be embodied in many different forms, and all such forms are intended to be within the scope of the invention. For each of the various aspects of the present invention, any such form of the above-described embodiment may be referred to herein as “logic configured to perform the described actions,” or alternatively, “logic” that performs the described actions.
[0029] As mentioned above, several techniques for facilitating one-handed operation of a device utilize user interface scaling, specifically scaling down the size of images displayed on the user's display screen to a size where the area is fully reachable by one finger (e.g., the user's thumb), and correspondingly scaling down the display's input touchpoints so that they fit entirely within the scaled-down area, allowing the user to select and interact with the displayed scaled-down image using the same finger. However, as also mentioned above, scaling down the size of images and input areas to enable one-finger access has its own associated problems and / or drawbacks, including the difficulty of reading and writing smaller content, and the fact that smaller buttons, and therefore related content, are often obscured (and thus invisible) by the finger being used.
[0030] To address these and / or related issues, in one embodiment consistent with the present invention, the technique involves utilizing screen space outside the scaled-down display area so that larger (e.g., full-size) images of at least some application icons or other information are displayed, and thus it is easier for the user to see (e.g., those images are not obscured by the user's fingers) and read (e.g., those images are larger than the scaled-down version of the information).
[0031] In other aspects of at least some embodiments consistent with the present invention, a mechanism such as user gaze detection is used to enable the user to interact with these items without requiring them to be able to reach unscaled information items on the display with one finger.
[0032] These and other embodiments will be described below.
[0033] Figures 1A, 1B, 1C, 1D, 1E, and 1F illustrate a device 101 having a touch-sensitive display 103, which is initially operated in full-size mode 105 and transitions to a scaled operating mode 107. The transition may be triggered by any of several known techniques, including, but not limited to, the detection of a predetermined gesture applied by the user to the touch-sensitive display. In this non-limiting example presented for illustrative purposes, the gesture includes first, second, third, and fourth touch points 109-1, 109-2, 109-3, and 109-4, respectively. Since all points within the gesture were reachable by the user's thumb while holding the device with one hand, all points within the scaled area 107 of the resulting touchscreen are also reachable to facilitate easy one-handed operation of the device 101.
[0034] Looking in more detail at the transition from full-size mode (where the user interacts with device 101 via the full-size area 105 of display 103) to scaled operation mode (where the user interacts with device 101 via a scaled area 107 of display 103), Figure 1A shows that the user is holding the mobile device in one hand and touching the screen with their thumb or other finger 119 at the first touch point 109-1. Device 101 is shown to be operated in full-size mode, where the entire touch-sensitive display 103 of the device (the full-size area 105 of the touch-sensitive display) is used. Furthermore, the size of any displayed item (e.g., a displayed image and / or displayed text) while in full-size mode is referred to herein as the reference size of that item. It will be understood that the reference size of an item is specific to that item. For example, a displayed photograph may occupy the entire full-size area 105 of display 103, and therefore its reference size is the size of the entire full-size area 105. In contrast, application icons displayed in full-size mode may occupy only a relatively small portion of the full-size area 105, and therefore their reference size is proportionally smaller than the reference size of the displayed photograph mentioned earlier.
[0035] As shown in Figure 1B, the user moves their thumb to the right and touches a second touch point 109-2 on the right edge of the screen and / or the edge of the device. This defines the starting point of the gesture and further identifies a first corner of the area of the touch-sensitive display where the scaled-down display window will be located.
[0036] Next, as shown in Figure 1C, the user continues to touch the screen while moving their thumb inward across the screen until the thumb settles at a third touch point 109-3. This defines a second corner of the area of the touch-sensitive display where the scaled-down display window will be located.
[0037] Next, as shown in Figure 1D, the user continues to touch the screen while moving their thumb downwards across the screen until the thumb stops at the fourth touch point 109-4. This defines the third corner of the area of the touch-sensitive display where the scaled-down display window is located. Note that there is no requirement that the fourth touch point 109-4 be at the bottom edge of the screen. Instead, the user may stop their thumb somewhere along the downward movement of the screen, and that stopping point is considered the fourth touch point 109-4.
[0038] In one embodiment consistent with the present invention, as shown in Figure 1E, the system infers from a supplied gesture that it is a mode change gesture and in response scales down the display to a scaled area 107 of the touchscreen with a configuration and size that has corners that fit within the area of the touch-sensitive display defined by the first, second, third, and fourth touch points 109-1, 109-2, 109-3, and 109-4.
[0039] Figure 1E further illustrates how the remaining interface area 113 of the touch-sensitive display 103, located outside the scaled area 107, remains after the scaling operation. According to one embodiment of the present invention, one or more information items 111 are displayed within the remaining interface area 113. As shown in the example in Figure 1E, these may include application icons, such as application icons 115. For easier reading by the user, it is advantageous to display them in a larger form (e.g., full size, or possibly unscaled size) than the corresponding application icons that may (but do not) be displayed within the scaled area 107.
[0040] Figure 1F similarly shows a user device 101 operating in scale mode, but in this non-limiting example, the information item 111 includes one or more text notifications 117 received and / or generated by device 201.
[0041] Figure 2 is a schematic diagram of several embodiments of the present invention. The user device 201 has a touch-sensitive display 203 that can be operated in full-size mode and scale mode. In the example of Figure 2, the device 201 is shown to be operated in scale mode and therefore has a scaled screen area 205 and the remaining interface area 207 located outside the scaled screen area 205. Scaled items, such as scaled application icons 213, are displayed within the scaled screen area 205. In one embodiment consistent with the present invention, each item displayed within the scaled screen area 205 has a corresponding scaled area size that is smaller than the corresponding reference size of that item.
[0042] It should be noted that both the positioning and size of the scaled screen area 205 are variable and, advantageously, can be configured by the user. For example, a left-handed user may configure the scaled screen area 205 to be positioned to the left of the touch-sensitive display 203 rather than to the right, as shown in the figure. Also, as mentioned above, the size of the scaled screen area 205 can, in some embodiments, be configured by the user so that all points within the scaled screen area 205 can be reached with one finger (for example, the user's thumb).
[0043] While the illustrated embodiments demonstrate operation in portrait mode, it should be further noted that aspects of the present invention also function well in devices that enable a scaling operation mode when the device is held and operated in landscape orientation.
[0044] Figure 2 also shows the presence of information items 209 displayed in the remaining interface area 207. The types of information items 209 include, but are not limited to, application icons 115 and other textual notices 117. In another aspect of embodiments consistent with the present invention, the size of any item displayed in the remaining interface area 207 (e.g., a displayed image and / or displayed text) has a related remaining area size that is larger than the scaled area size associated with that item. In some embodiments, but not all, the remaining area size of an item may be equal to its base size, but this is not an essential aspect of embodiments of the present invention. Rather, the remaining area size of any item can be any size that is larger than the scaled area size of that item.
[0045] These display configurations make it easier for the user to interact with the device 201 because important information (e.g., application icons, notifications, etc.) is displayed in a position where it is not obscured by the user's fingers 119, and furthermore, important information is displayed in a size larger than the scaled area size of the displayed information (e.g., unscaled).
[0046] As described above, further embodiments of the present invention enable the user to interact with information items 209 without requiring them to reach the information items 209 with one finger 119 (which can be extremely difficult with one hand, as the reachable points are set to be mostly contained within the scaled screen area 205). In this exemplary embodiment, interaction with any of the information items 209 is facilitated by the use of a user-facing camera 211 located within the user device 201. The camera 211 senses the user's image and provides this information to a processing circuit contained within the device 101. The processing circuit, by analyzing the image data, detects when the user is gazing at one of the information items 209. In response, the processing causes a predetermined action to be taken with respect to a particular one of the information items 209 presented within the remaining interface area 207. The prescribed actions include, but are not limited to, highlighting the image of the detected information item to indicate its selection (for example, highlighting the application icon 115 with the relevant remaining area size), and / or launching the application corresponding to the first information item (for example, launching the application corresponding to the application icon 115).
[0047] In some embodiments consistent with the present invention, though not necessarily all, the user may interact with the device in ways other than those involving the use of a single finger capable of reaching the scaled area 107. For example, shaking and / or tilting the device in a predetermined manner may be detected by an inertial measuring unit (IMU) contained within the device 201, and this movement may, for example, cause one of the information items 209 to be selected and / or highlighted. In another embodiment, further tilting / shaking may be used to change the selection of a highlighted information item from the currently selected / highlighted information item to another information item. This feature is particularly useful when the user has gazed at icons but has not selected any particular one of those icons. In such an example, tilting / shaking is an easy way for the user to indicate that they would like to see a new set of icons from which they may make a selection. In another embodiment, if the user stops gazing at icons (either directly or after viewing a set of several icons) without selecting a particular icon, the displayed group of icons may be reset to the original set.
[0048] The device behavior described above can be initiated by the same trigger that operates the device in scale mode. When such operation is initiated, the system informs system applications and user applications about the available unused screen area located outside the scaled screen area 205 (i.e., the remaining interface area 207), which can then be used in the default manner set by the system as configured at the device factory, or in a manner modified and customized by the user (and stored in user settings). A useful but not mandatory default is to use the remaining interface area 207 for application switching by displaying application icons 115 therein, and to allow the user to switch between applications by means of gaze tracking as described above. It should be noted that using gaze tracking in this way is much easier with respect to information items 209 having their respective remaining area sizes than for scaled corresponding items displayed in the scaled screen area 205, because larger items are much easier to track and require less precision in processing.
[0049] In some, though not all, embodiments of the invention, a gesture in the form of a user glancing at one of the information items 209 (for example, for a predetermined amount of time), followed by a physical movement of the device 201 (for example, tilting or shaking the device), triggers the selection / highlighting of one of the information items 209. For example, a brief glance by the user, followed by shaking or tilting the device 201, triggers the selection / highlighting of the information item the user is fixated on. In another embodiment, re-selection from among the information items may be triggered by further shaking or tilting the device 201.
[0050] While not all, alternative embodiments of some of the inventions relate to precisely determining which information items 209 should be displayed in the remaining interface area 207 when the scale mode is first activated. Several alternative forms are possible, including the following:
[0051] The device circuit detects that a scaled application icon currently displayed in the scaled area 205 of the touch-sensitive display 203 is covered by the user's finger 119 on the touch-sensitive display 203, and in response to this detection, displays a larger (remaining area size) application icon 115 as one of the information items 209 displayed in the remaining interface area 207, with the remaining area size application icon 115 corresponding to the scaled area size application icon.
[0052] In one alternative embodiment, in response to detecting that an area of a scaled area 205 of the touch-sensitive display 203 is covered by the user's finger 119 on the touch-sensitive display 203, the device 201 displays a rendering of the remaining area size of the scaled area-size image displayed within the scaled area of the touch-sensitive display as one of one or more information items 209 displayed within the remaining interface area of the touch-sensitive display (for example, displaying a reference-size rendering of a portion of a map, photograph, etc.). In one variation of this embodiment, the position of the user's finger 119 is indicated on the remaining area-size image (for example, a round red marker), thereby facilitating the user to assess what the finger 119 is currently covering / blocking in the scaled-down screen area 205.
[0053] In another alternative configuration, when entering a scaling operation mode, the device detects which of the multiple applications was most recently activated when the user interface was operated in full-size mode, selects the application icon corresponding to the detected most recently activated application, and displays the selected application icon as one or more information items 209.
[0054] In yet another alternative form, upon entering a scaling operation mode, the device detects which of several applications was most frequently activated when the user interface was operated in full-size mode, selects the application icon corresponding to the detected most frequently activated application, and displays the selected application icon as one or more information items 209. In this variant of the embodiment, detecting the most frequently activated application is constrained to detecting which of several applications was most frequently activated during a predetermined number of recent time periods when the user interface was operated in full-size mode.
[0055] In yet another alternative configuration, once the device enters a scaled operating mode, the remaining interface area 207 can be used to display text notifications 117 received and / or generated by the device.
[0056] Next, exemplary embodiments consistent with the present invention will be further described with reference to Figure 3, which in one view is a flowchart of the actions performed by devices 101, 201 to transition their operating mode from full-size interface mode to scale mode, and the actions performed when operating in scale mode. In another view, the blocks shown in Figure 3 can also be considered to represent means 300 (e.g., hardwired or programmable circuits or other processing means) for performing the described actions.
[0057] In step 301, devices 101 and 201 are operating in full-size mode and remain in this mode unless a predetermined mode change gesture is detected (the "no" path from judgment block 303). When the input of a predetermined mode change gesture is detected (the "yes" path from judgment block 303), the devices respond by transitioning to a scaled operating mode (step 305). Unless an implementation-specific mode change indicator is detected (the "no" path from judgment block 307), devices 101 and 201 remain in the scaled operating mode. However, when a mode change indicator is detected (the "yes" path from judgment block 307), devices 101 and 201 return to step 301 and resume full-size operating mode.
[0058] Operation in full-size mode is well known in the art and therefore does not need to be described in detail. In short, it involves displaying an output image across a first area 105 of the touch-sensitive displays 103, 203 (typically, the first area encompasses the entire screen area of the touch-sensitive display or nearly it) and accepting touch input from an application or system software at a set of input touch points within the first area 105 of the touch-sensitive displays 103, 203. Items displayed to the user within the first area 105 are shown in their respective reference sizes.
[0059] Operating in scale mode (step 305) involves scaling the output image and displaying the scaled output image on the scaled areas 107, 205 of the touch-sensitive display 103 (step 307), where the scaled areas are smaller than the first area 105. This is illustrated in Figures 1E, 1F, and 2, respectively. A set of input touchpoints in full-size mode is scaled so that the set of scaled input touchpoints lies within the scaled areas 107, 205 of the touch-sensitive display 103.
[0060] Furthermore, according to embodiments consistent with the present invention, one or more information items 111, 209, including the first information items 115, 117, are displayed in a format larger than the scaled area size within the remaining interface area 207 of the touch-sensitive displays 103, 203 (step 311), the remaining interface area 207 being part of the touch-sensitive displays 103, 203 located outside the scaled area 107, 205 of the touch-sensitive displays 103, 203.
[0061] To facilitate the user's ability to interact with the information items 111 and 209 displayed in the remaining interface area 207, the scaled operation mode further includes acquiring user image data (step 313). This may be done by a user-facing camera, such as camera 211 shown in Figure 2. The image data is analyzed using known methods (step 315), and from the results of the analysis, it is detected when the user is gazing at the first information items 111 and 209 in the remaining interface area 207 (determination block 317). If no such gaze is detected (the "no" path from determination block 317), devices 101 and 201 continue to operate in scaled mode and perform the actions described above. In response to detecting that the user has gazed at one of the information items 111 and 211 (the "yes" path from determination block 317), devices 101 and 201 display a highlighted image of the first information item and / or activate the application corresponding to the first information item (step 321). (Note: The actions shown by dashed lines in Figure 3 are optional and therefore not required in all embodiments. Thus, in some embodiments, though not all, a direct path exists between the determination block 317 and step 321.)
[0062] In some, though not all, alternative embodiments, in response to detecting that the user is gazing at the first information items 111, 209 in the remaining interface area 207 (the "yes" path from determination block 317), the device checks whether a given device movement (e.g., shaking or tilting) followed the gaze (determination block 319). If not (the "no" path from determination block 319), the process proceeds to step 321, where the action described above is performed.
[0063] However, if it is detected that the movement of the specified device continued after the gaze (the "yes" path from determination block 319), the gaze is interpreted differently, for example, by changing which of information items 111 or 209 is displayed in a highlighted format (i.e., to indicate that it was selected in some way) (step 323). The process then proceeds to determination block 307, where the action described above is performed.
[0064] As will be described in various embodiments, Figure 4 shows exemplary configurations of a controller 401 that may be included in devices 101, 201 to perform any and / or all of the actions described above, and Figure 4 shows exemplary controllers 401 of devices 101, 201 in several exemplary embodiments consistent with the present invention, though not necessarily all. In particular, the controller 401 includes circuitry configured to perform any one or any combination of the various functions described above. Such circuitry may be, for example, a complete hardwired circuit (e.g., one or more application-specific integrated circuits – “ASICs”). However, the exemplary embodiment in Figure 4 shows a programmable circuitry including a processor 403 coupled to one or more memory devices 405 (e.g., random access memory, magnetic disk drive, optical disk drive, read-only memory, etc.) and an interface 407 that enables bidirectional communication with other elements of devices 101, 201. A complete list of other conceivable elements is beyond the scope of this specification. For the purposes of illustrating this configuration, the device is shown to include both a user-facing camera 411 and an inertial measurement unit 413. The user-facing camera 411 senses the user's image, which can be used to detect when the user is fixating on one of the information items 111, 209, as described above. The inertial measurement unit 413 can be used to generate sensing data indicating whether the device is tilted and / or shaken, and actions are taken based on these indications, as described above.
[0065] The memory device 405 stores programming means 409 (e.g., sets of processor instructions) configured to cause the processor 403 to control other system elements in order to perform any of the embodiments described above. The memory device 405 may also store data (not shown) representing various constants and variable parameters that may be required by the processor 403 and / or specified by the programming means 409 when performing its functions.
[0066] Embodiments consistent with the present invention offer several advantages over previously known technologies. These advantages include simplified one-handed operation on mobile devices, easier readability of important information when the device is operated in one-handed mode, and an easier way to input information about displayed items that are out of reach of the fingers of a user operating a scaled-down version of the display.
[0067] The present invention has been described with reference to specific embodiments. However, it will be apparent to those skilled in the art that the present invention can be carried out in specific forms other than those described above. Therefore, the embodiments described are illustrative and should not be considered limiting in any way. The scope of the present invention is further illustrated not only by the foregoing description but also by the appended claims, and all variations and equivalents included within the claims are intended to be encompassed therein.
Claims
1. A method for operating the user interface of a device (101, 201), wherein the user interface includes touch-sensitive displays (103, 203), and the method is While operating the devices (101, 201) in full-size mode (301), it is detected (303) that a mode change gesture has been received by the touch-sensitive displays (103, 203), In response to the detection, the user interface is operated in scale mode (305) Including when an application or system software generates an output image to be displayed on the device (101, 201), Operating the user interface in the full-size mode (301) is, Displaying the output image on the first area (105) of the touch-sensitive display (103, 203), This includes accepting touch input from the application or system software at a set of input touch points located within the first area (105) of the touch-sensitive display (103, 203), Operating the user interface in the aforementioned scale mode (305) means 307) scaling the output image and displaying the scaled output image on the scaled area (107, 205) of the touch-sensitive display (103, 203), wherein the scaled area (107, 205) is smaller than the first area (105). Scaling the set of input touch points (309) such that the scaled set of input touch points lies within the scaled area (107, 205) of the touch-sensitive display (103, 203), Displaying one or more information items (111, 209), including a first information item (115, 117), within the remaining interface area (113, 207) of the touch-sensitive display (103, 203) (311), wherein the remaining interface area (113, 207) is a part of the touch-sensitive display (103, 203) located outside the scaled area (107, 205) of the touch-sensitive display (103, 203), and displaying the information items therein. Obtaining user image data (313) Includes: analyzing the image data (315) to detect (317) when the user is gazing at the first information items (115, 117) in the remaining interface areas (113, 207); and in response to the detection, displaying a highlighted image of the first information items (115, 117) and / or activating the application corresponding to the first information items (115, 117) (321). The aforementioned method, The detection of the scaled application icon (213) currently displayed in the scaled area (107, 205) of the touch-sensitive display (103, 203) being covered by the user's finger (119) on the touch-sensitive display (103, 203), A method for operating the user interface of a device (101, 201), further comprising displaying an unscaled application icon (115) as one of the one or more information items (111, 209) displayed in the remaining interface area (113, 207) of the touch-sensitive display (103, 203), in response to detecting that the scaled application icon (213) is covered by the user's finger (119) on the touch-sensitive display (103, 203), wherein the unscaled application icon (115) displays an application icon (115) corresponding to the scaled application icon (213).
2. A method for operating the user interface of a device (101, 201), wherein the user interface includes touch-sensitive displays (103, 203), and the method is While operating the devices (101, 201) in full-size mode (301), it is detected (303) that a mode change gesture has been received by the touch-sensitive displays (103, 203), In response to the detection, the user interface is operated in scale mode (305) Including when an application or system software generates an output image to be displayed on the device (101, 201), Operating the user interface in the full-size mode (301) is, Displaying the output image on the first area (105) of the touch-sensitive display (103, 203), This includes accepting touch input from the application or system software at a set of input touch points located within the first area (105) of the touch-sensitive display (103, 203), Operating the user interface in the aforementioned scale mode (305) means 307) scaling the output image and displaying the scaled output image on the scaled area (107, 205) of the touch-sensitive display (103, 203), wherein the scaled area (107, 205) is smaller than the first area (105). Scaling the set of input touch points (309) such that the scaled set of input touch points lies within the scaled area (107, 205) of the touch-sensitive display (103, 203), Displaying one or more information items (111, 209), including a first information item (115, 117), within the remaining interface area (113, 207) of the touch-sensitive display (103, 203) (311), wherein the remaining interface area (113, 207) is a part of the touch-sensitive display (103, 203) located outside the scaled area (107, 205) of the touch-sensitive display (103, 203), and displaying the information items therein. Obtaining user image data (313) The analysis of the image data (315) detects (317) when the user is gazing at the first information items (115, 117) in the remaining interface areas (113, 207), and in response to the detection, displays a highlighted image of the first information items (115, 117) and / or activates an application corresponding to the first information items (115, 117) (321), The aforementioned method, To detect that a portion of the scaled area (107, 205) of the touch-sensitive display (103, 203) is covered by the user's finger (119) on the touch-sensitive display (103, 203), A method for operating the user interface of a device (101, 201), further comprising: detecting that the area of the scaled area (107, 205) of the touch-sensitive display (103, 203) is covered by the user's finger (119) on the touch-sensitive display (103, 203); and displaying an unscaled rendering of the scaled image displayed in the area of the scaled area (107, 205) of the touch-sensitive display (103, 203) as one of the one or more information items (111, 209) displayed in the remaining interface area (113, 207) of the touch-sensitive display (103, 203).
3. A method for operating the user interface of a device (101, 201), wherein the user interface includes touch-sensitive displays (103, 203), and the method is While operating the devices (101, 201) in full-size mode (301), it is detected (303) that a mode change gesture has been received by the touch-sensitive displays (103, 203), In response to the detection, the user interface is operated in scale mode (305) Including when an application or system software generates an output image to be displayed on the device (101, 201), Operating the user interface in the full-size mode (301) is, Displaying the output image on the first area (105) of the touch-sensitive display (103, 203), This includes accepting touch input from the application or system software at a set of input touch points located within the first area (105) of the touch-sensitive display (103, 203), Operating the user interface in the aforementioned scale mode (305) 307) scaling the output image and displaying the scaled output image on the scaled area (107, 205) of the touch-sensitive display (103, 203), wherein the scaled area (107, 205) is smaller than the first area (105). Scaling the set of input touch points (309) such that the scaled set of input touch points lies within the scaled area (107, 205) of the touch-sensitive display (103, 203), Displaying one or more information items (111, 209), including a first information item (115, 117), within the remaining interface area (113, 207) of the touch-sensitive display (103, 203) (311), wherein the remaining interface area (113, 207) is a part of the touch-sensitive display (103, 203) located outside the scaled area (107, 205) of the touch-sensitive display (103, 203), and displaying the information items therein. Obtaining user image data (313) The analysis of the image data (315) detects (317) when the user is gazing at the first information items (115, 117) in the remaining interface areas (113, 207), and in response to the detection, displays a highlighted image of the first information items (115, 117) and / or activates an application corresponding to the first information items (115, 117) (321), The aforementioned method, A method for operating the user interface of a device (101, 201), further comprising detecting that the user gazes at one or more information items (111, 209) within the remaining interface areas (113, 207) for a predetermined amount of time or less, and then shakes the device (101, 201), and in response to the detection, displaying a highlighted image of the information item (111, 209).
4. Operating the user interface in the aforementioned scale mode means that The method according to any one of claims 1 to 3, further comprising displaying an image generated by the application within the scaled area (107, 205) of the touch-sensitive display (103, 203) when the application corresponding to the first information item (115, 117) is activated.
5. The first information item (115, 117) is one of a plurality of information items (111, 209) that are displayed in the remaining interface area (113, 207) of the touch-sensitive display when the user interface is operated in the scale mode. The method according to any one of claims 1 to 4, comprising: detecting from the analysis of the image data which of the plurality of information items (111, 209) the user is fixated on; displaying a highlighted image of the detected information item among the information items (111, 209); and / or activating an application corresponding to the detected information item among the information items (111, 209).
6. Operating the user interface in the aforementioned scale mode means that When the user interface is operated in full-size mode, it is possible to detect which of the multiple applications was most recently activated, Select an application icon (115) corresponding to the detected application from among the plurality of applications, and display the selected application icon (115) as one of the one or more information items (111, 209) in the remaining interface area (113, 207) of the touch-sensitive display (103, 203). The method according to any one of claims 1 to 5, including the method described in any one of claims 1 to 5.
7. Operating the user interface in the aforementioned scale mode means that When the user interface is operated in full-size mode, it is possible to detect which of the multiple applications was the most frequently activated application. Select the application icon (115) corresponding to the detected application from among the multiple applications that was the most frequently activated application, and display the selected application icon (115) as one of the one or more information items (111, 209) in the remaining interface area (113, 207) of the touch-sensitive display (103, 203). The method according to any one of claims 1 to 6, including
8. The method according to claim 7, wherein detecting which of the plurality of applications was the most frequently activated application when the user interface was operated in full-size mode includes detecting which of the plurality of applications was the most frequently activated application in the most recent predetermined number of times the user interface was operated in full-size mode.
9. The method according to any one of claims 1 to 8, wherein at least one of the one or more information items (111, 209) is a text notification (117) received and / or generated by the device (101, 201).
10. The method according to claim 3, comprising detecting that the user tilts the device (101, 201) in response to replacing the highlighted image of the information item (111, 209) with an unhighlighted image of the information item (111, 209), and displaying a highlighted image of a different information item (111, 209) from the one or more information items (111, 209).
11. A computer program (409) including instructions, wherein when the instructions are executed by at least one processor (403), the computer program (409) causes the at least one processor (403) to perform the method according to any one of claims 1 to 10.
12. An apparatus for operating the user interface of a device (101, 201), wherein the user interface includes touch-sensitive displays (103, 203), and the apparatus includes one or more circuits configured to perform the method according to any one of claims 1 to 10.