Modeless enhancements to the virtual trackpad on multi-screen computing devices

Abstract

The disclosed technology solves various technical and user experience problems by enhancing the functionality provided by a virtual on-screen trackpad without requiring the user to switch between modes. In this way, the user can access the extended functionality without interfering with the expected traditional trackpad functionality (e.g., moving the cursor, clicking, and scrolling). In particular, the following technologies are disclosed: modeless digital pen input to a virtual trackpad, modeless gestures for summoning user interfaces, modeless gestures utilizing positions relative to adjacent user interfaces, and modeless cursor control and interaction with virtual touch targets.

Description

Background Art

[0001] A trackpad (which may also be referred to as a touchpad) is a user input pointing device with a dedicated flat surface capable of detecting finger contact. The surface can translate the position and motion of one or more fingers of a user into relative positions on the screen of a computing device. The tracked positions and motions of the user's fingers can then be used to move an on-screen cursor or perform other types of functionality. A trackpad also includes functionality for performing activation operations (such as by clicking a physical button or tapping the trackpad). However, the functionality provided by conventional trackpads is typically limited to cursor control, selection, and potentially basic single-touch or multi-touch gestures.

[0002] A virtual on-screen trackpad enables a touch screen display to provide functionality similar to a traditional physical trackpad. However, existing virtual on-screen trackpads typically only replicate (usually poorly) the behavior of a traditional trackpad. Moreover, in addition to other functionality, the on-screen region that supports traditional trackpad functions also relies on the user to perform an explicit mode switch, which requires additional steps and takes the user out of their workflow. The user input required to perform a mode switch may also be difficult to remember and confusing to the user, resulting in inadvertent or incorrect user input, which unnecessarily consumes computing resources, such as processor cycles and memory. Performing a mode switch may also result in the execution of additional program code, which also consumes computing resources, such as processor cycles and memory.

[0003] It is with respect to these and other technical challenges that the disclosure made herein is presented. Summary of the invention

[0004] Disclosed herein is a technology that can enhance the functionality provided by a virtual touchpad modelessly. The disclosed technology solves the technical problems discussed above by enhancing the functionality provided by a virtual on-screen touchpad without requiring the user to switch between modes. In this way, the user can access extended functionality without interfering with the expected traditional touchpad functionality (e.g., moving the cursor, clicking, and scrolling), thereby having an improved user experience. Additionally, compared to previous mode virtual touchpad solutions that require mode switching, the utilization of computing resources can be reduced by simplified user interaction and less execution of program code. Other technical benefits not specifically mentioned herein can also be achieved by implementation of the disclosed subject matter.

[0005] In some configurations, the techniques disclosed herein are implemented in conjunction with a multi-screen computing device. Multi-screen computing devices come in a variety of form factors, such as a two-screen articulated device similar to a traditional laptop computer. A device configured in this manner includes a first display device that is located in the same position as a display on a traditional laptop computer. In various configurations, the first display device may or may not be touch sensitive. The first display device provides a first display area that occupies all or a portion of a physical display.

[0006] In some configurations of multi-screen computing devices, a second display device is located in the position where a physical keyboard is traditionally located on a laptop computer. The second display device may occupy the entire area where a physical keyboard is traditionally located or may occupy only a portion of the area. The second display device is equipped with a sensor for detecting user input (such as a single-touch or multi-touch gesture). The second display device provides a second display area that occupies all or a portion of the physical display.

[0007] An edge (e.g., a bottom edge) of the first display area is arranged adjacent to and parallel to an edge (e.g., a top edge) of the second display area. The distance between the two display areas can be larger or smaller, depending on the specific mechanical connection (e.g., a hinge) of the respective housings connecting the display devices providing the display areas. The respective sides of the first display area and the second display area can also be spatially aligned. As will be described in more detail below, the spatial alignment of the display areas enables certain types of modeless gestures to be performed in the second display area relative to corresponding intervals of the first display area.

[0008] The second display area can be divided into a virtual touchpad interval and a virtual keyboard interval. The virtual touchpad interval can be used as a virtual touchpad. For example, the virtual touchpad can determine the position and movement of the user's finger, which can then be used to move the cursor on the screen or perform other types of functionality. The virtual touchpad interval can also be used to perform activation operations, such as, for example, by tapping within the virtual touchpad interval. The virtual touchpad interval encompasses the entire width of the second display area, and in some configurations, is located in a portion of the second display area adjacent to the first display area.

[0009] Modeless digital pen input to virtual trackpad

[0010] In one configuration, the virtual touchpad section can also receive input from a digital pen without requiring the user to change the operating mode of the virtual touchpad. In this configuration, the user can write directly in the virtual touchpad section using a digital pen at any time. In particular, when input is received in the virtual touchpad section, the computing device determines whether the input is touch input (i.e., the user's finger) or input from the digital pen. For example, if the input is touch input, the input is processed as touch input to the virtual touchpad to move the cursor or perform an activation operation. If the input is input received from a digital pen, the input is processed as digital ink.

[0011] Processing the input as digital ink may include, but is not limited to, converting the digital ink into text and providing the text to a program so as to present the text within a field of a user interface ("UI") presented in a first display area. Processing the input as digital ink may also include providing the digital ink to a program so as to present the digital ink within a field of the UI presented in a second display area. Processing the input as digital ink may also include converting the digital ink into text, identifying commands in the text, and causing the computing device to execute the commands. Processing the input as digital ink may additionally include providing the digital ink to a default program for notes, or converting the digital ink into text and providing the text to a default program for storing notes. In some configurations, it is not necessary to convert the digital ink into text. Instead, commands may be identified directly from the digital ink, such as, for example, when the ink represents non-text information (such as a shape or arrow).

[0012] Modeless gesture for summoning the user interface

[0013] In another configuration, the virtual touchpad interval can be utilized to initiate presentation of a transient UI in the second display area for executing commands or viewing information without requiring the user to change the operating mode of the virtual touchpad interval. Specifically, when receiving input within the virtual touchpad interval, the computing device determines whether the touch input includes a touch gesture that originates outside the edge of the virtual touchpad interval of the second display area and terminates inside the virtual touchpad interval. For example, the user can swipe from a location outside the virtual touchpad interval to a location within the virtual touchpad interval using one or more fingers.

[0014] If the computing device detects a touch gesture that originates outside the edge of the virtual trackpad interval of the second display area and ends inside the virtual trackpad interval, the computing device can display the transient UI in the virtual trackpad interval. The transient UI can be animated when it is displayed, such as, for example, by "sliding" from the edge of the virtual trackpad interval where the gesture originates into the virtual trackpad interval.

[0015] In some configurations, the transient UI includes selectable UI elements, such as UI buttons or other types of UI elements. If the computing device determines that a touch input selecting one of the one or more selectable UI elements has been performed in the virtual trackpad interval, the computing device can initiate a command corresponding to the selected UI element. For example, a command can be initiated to launch an application that can present a UI in the first display area, switch to an already launched application (i.e., switch tasks), or perform another function.

[0016] In some configurations, in response to selection of one of the UI elements, the display of the transient UI is removed from the virtual trackpad interval. In other configurations, the transient UI can be dismissed using a touch gesture. For example, a user can swipe from a location inside the virtual trackpad interval to a location outside the virtual trackpad interval using one or more fingers to dismiss the transient UI. The display of the transient UI can also be animated when the transient UI is removed, such as, for example, by "sliding" the transient UI outside the virtual trackpad interval to the edge of the virtual trackpad interval where the gesture terminates.

[0017] In other configurations, an input gesture made from outside the edge of the virtual trackpad interval to a position inside the virtual trackpad interval can perform a function based on the relationship between the end position of the gesture and a UI element presented in the keyboard interval in the first display area or the second display area. For example, a user can set the output volume of the computing device by swiping right from an interval outside the virtual trackpad interval to a position inside the virtual trackpad interval. The position where the gesture ends (e.g., horizontally aligned) relative to the virtual keyboard specifies the output volume. For example, if the gesture ends at a position that is spatially aligned with the '1' key on the virtual keyboard, the output volume will be set to one (e.g., 10% of the maximum value). If the gesture ends at a position that is spatially aligned (e.g., horizontally aligned) with the '9' key on the virtual keyboard, the output volume will be set to nine (e.g., 90% of the maximum value). In this way, many different specific edge gestures can be performed along a common edge of the virtual trackpad interval.

[0018] Modeless gestures using relative position

[0019] In another configuration, a virtual trackpad interval can be utilized to enable a gesture having functionality determined based on a starting point of the gesture relative to an adjacent on-screen UI, another type of object, or a keyboard. In this configuration, a first application executed on a computing device can present a UI in a first portion of a first display area: for example, a left half of the first display area. The first portion of the virtual trackpad interval is spatially aligned with the first portion of the first display area. For example, the first portion of the virtual trackpad interval can include a left half of the second display area that is spatially aligned with a left half of the first display area.

[0020] Similarly, a second application executed on the computing device can present a UI in a second portion of the first display area: for example, the right half of the first display area. The second portion of the virtual trackpad interval is spatially aligned with the second portion of the first display area. For example, the second portion of the virtual trackpad interval can include the right half of the second display area, which is spatially aligned with the right half of the first display area. In this way, one application is assigned a portion of the first display area corresponding to the portion of the virtual trackpad interval that is spatially aligned, and another application is assigned another portion of the first display area that is spatially aligned with another portion of the virtual trackpad interval.

[0021] If the computing device receives input in the first portion of the virtual trackpad interval, the computing device provides the input to the application, or interprets the input (e.g., a command) relative to the application whose UI is being presented in the adjacent portion of the first display area. For example, if the input is received in the left half of the virtual trackpad interval, the input will be provided to the application whose UI is being presented in the left half of the first display area. If the computing device receives input in the second portion of the virtual trackpad interval, the computing device provides the input to the application whose UI is being presented in the adjacent portion of the first display area. For example, if the input is received in the right half of the virtual trackpad interval, the input will be provided to the application whose UI is being presented in the right half of the first display area. The input can be, for example, a multi-touch input gesture or input received from a digital pen, so as to distinguish the input from a traditional virtual trackpad input for controlling a cursor, for example. Specific examples of gestures include gestures for minimizing or closing windows associated with an application.

[0022] In some configurations, a computing device may detect a user input gesture that originates in an interval outside of a virtual trackpad interval and terminates within the virtual trackpad interval. For example, a user input gesture, such as a swipe gesture, may originate within a keyboard interval and terminate within a virtual trackpad interval. In response to detecting such a user input gesture, the computing device may execute a command selected based at least in part on the location where the gesture originated. For example, when a gesture originates from a key of a virtual keyboard and ends in a virtual trackpad interval, a command may be selected based on the specific key from which the gesture originated.

[0023] Modeless cursor control and interaction with touch targets

[0024] In another configuration, a display area surrounding a virtual trackpad interval on a touch-sensitive display may show touch targets for initiating various types of functionality. Cursor control may be performed using the virtual trackpad interval, and the touch targets may be selected without changing the mode of the virtual trackpad. For example, one or more UI controls may be displayed outside the virtual trackpad interval. When a UI control is selected, such as by using a touch gesture, the UI control may initiate various types of functionality, such as, for example, launching a program or summoning a digital assistant.

[0025] In this configuration, a touch gesture can originate from within a virtual touchpad interval and terminate within one of the UI controls without causing selection of the UI control. In this way, inadvertent exiting of the virtual touchpad interval when performing a cursor control gesture will not cause selection of one of the UI controls. Similarly, a touch gesture can originate from within one of the UI controls and terminate within the virtual touchpad interval or another interval without causing selection of the UI control. In this way, even if a cursor control gesture starts within a UI control, it can be executed outside the virtual touchpad interval. The starting position and timing of the user input gesture can be used to disambiguate cursor control and interaction with the touch target without the need for mode switching.

[0026] In some configurations, the computing device can detect a user input gesture that originates within the virtual trackpad interval and terminates outside the virtual trackpad interval. In this example, the computing device can execute a command selected based at least in part on the object at which the gesture terminates.

[0027] In another configuration, a physical input object may be placed on one of several areas of the virtual trackpad zone to enable other functionality, such as, but not limited to, providing direct control of the volume, brightness, and scrolling of the computing device. For example, a numeric dial or other type of object may be placed in the virtual trackpad zone and manipulated to adjust settings such as these and perform other types of functionality.

[0028] Although the embodiments disclosed herein are primarily presented in the context of a multi-screen computing device, the disclosed technology can also be implemented in conjunction with a single-screen computing device that utilizes a single display screen to provide a first display area and a second display area. It should also be understood that although the embodiments briefly described above are generally described separately, the various embodiments briefly described above and described in further detail below can be utilized in combination with each other.

[0029] It should also be understood that the subject matter described above can be implemented as a computer-controlled apparatus, a computer-implemented method, a computing device, or an article such as a computer-readable medium. These and various other features will become apparent by reading the following detailed description and referring to the associated drawings.

[0030] The present summary is provided to introduce in a simplified form a brief description of some aspects of the disclosed technology that are further described below in the detailed description. The present summary is neither intended to identify key features or essential features of the claimed subject matter nor to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic diagram illustrating aspects of the configuration and operation of a multi-screen computing device implementing the disclosed technology in one particular configuration;

[0032] Figures 2 to 5 is a schematic diagram illustrating aspects of the mechanism disclosed herein for modeless digital pen input on a virtual trackpad;

[0033] Figure 6 is a flowchart showing a routine that illustrates Figure 1 The computing device shown in FIG. Figures 2 to 5 Aspects of the operation of a modeless digital pen input on a virtual trackpad as shown in;

[0034] Figures 7 to 11 is a schematic diagram illustrating aspects of a mechanism disclosed herein for modelessly summoning a user interface in a virtual trackpad interval;

[0035] Fig.12 is a flowchart showing a routine that illustrates Figure 1 The computing device shown in FIG. Figures 7 to 11 Aspects of the operation of the user interface in the virtual touchpad section shown in ;

[0036] Fig.13 and Fig.14 is a schematic diagram illustrating aspects of a mechanism disclosed herein for enabling modeless gestures on a virtual trackpad, the modeless gestures having functionality determined based on the location of the gesture's origin relative to an on-screen UI or physical keyboard;

[0037] Fig.15 is a flowchart showing a routine that illustrates Figure 1 Aspects of the operation of a computing device for enabling a modeless gesture on a virtual trackpad, the modeless gesture having a gesture-based gesture start point relative to a touch pad, Fig.13 and Fig.14 The functionality is determined by the position of the UI on the screen shown in ;

[0038] Figures 16 to 19 is a schematic diagram illustrating aspects of the mechanisms disclosed herein for enabling modeless cursor control and interaction with virtual touch targets;

[0039] Fig. 20 is a flowchart showing a routine that illustrates Figure 1 The computing device shown in FIG. Figures 16 to 19 Aspects of the operation of modeless cursor control and interaction with virtual touch targets as shown in ; and

[0040] Fig.21 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing device that can implement aspects of the techniques presented herein. DETAILED DESCRIPTION

[0041] The following detailed description relates to techniques for modelessly enhancing the functionality provided by a virtual touchpad. As briefly discussed above, implementation of the disclosed techniques can enable users to access extended functionality without interfering with expected traditional touchpad functionality (e.g., moving the cursor, clicking, and scrolling). Therefore, compared to previous mode virtual touchpad solutions, utilization of computing resources can be reduced by simplified user interaction and execution of less program code. Other technical benefits not specifically mentioned herein can also be achieved by implementation of the disclosed subject matter.

[0042] Those skilled in the art will recognize that the subject matter disclosed herein can be implemented with various types of computing systems and modules, at least some of which are described in more detail below. Those skilled in the art will also appreciate that the subject matter described herein can be practiced with other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, computing or processing systems embedded in devices such as wearable computing devices, automobiles, home automation, etc., computing or processing systems embedded in devices such as wearable computing devices, automobiles, home automation, etc.

[0043] In the following detailed description, reference is made to the accompanying drawings that form a part hereof and that show specific configurations or examples by way of illustration. Referring now to the drawings, in which like reference numerals represent like elements throughout the several figures, aspects of various techniques for modelessly enhancing the functionality provided by a virtual trackpad will be described.

[0044] Figure 1 is a schematic diagram illustrating aspects of the configuration and operation of a multi-screen computing device 100 (which may also be referred to herein as “computing device 100” or simply “device 100”) that implements the disclosed technology in one particular configuration. As mentioned above, multi-screen computing devices (such as computing device 100) come in a variety of form factors. For example, Figure 1 The computing device 100 shown in the figure is configured as a dual-screen articulated device similar to a traditional laptop computer. However, the disclosed technology can also be utilized with multi-screen computing devices having other configurations. As discussed above, the disclosed technology can also be practiced with a computing device having a single folding display screen, such as a computing device utilizing flexible screen technology.

[0045] like Figure 1 As shown, the illustrative computing device 100 includes two display devices. The first display device is mounted in a housing 104A and is located in the same position as on a traditional laptop computer when in use. The first display device provides a first display area 102A that surrounds all or a portion of the physical display. The first display device may or may not be touch sensitive.

[0046] The second display device is located in the area where the physical keyboard is traditionally located on a laptop computer and provides a second display area 102B. The second display area may occupy the entire area where the physical keyboard is traditionally located or may occupy only a portion of the area. The second display device is equipped with a sensor for detecting user input (such as a single-touch or multi-touch gesture).

[0047] exist Figure 1In the example configuration shown in , the first display area 102A has a height 106A. The second display area 102B has a height 106B, which may or may not be the same as the height 106A. The first display area 102A also has a width 108A, and the second display area 102B has a width 108B. In some configurations, the width 108B of the second display area 102B is the same as the width 108A of the first display area 102A. In other configurations, the width 108B of the second display area 102B may be greater than or less than the width 108A of the first display area 102A.

[0048] exist Figure 1 In the configuration shown in , the bottom edge of the first display area 102A is parallel to the top edge of the second display area 102B. The bottom edge of the first display area 102A is also adjacent to the top edge of the second display area 102B. Figure 1 In the example shown in , the housing 104A of the first display area 102A is separated from the housing 104B of the second display area by a distance 114. The distance 114 may be larger or smaller depending on the specific mechanical connection (e.g., hinge) connecting the housing 104A and the housing 104B. In the case where the two display areas are part of the same physical, potentially flexible display screen, the distance 114 may also be zero.

[0049] exist Figure 1 , the left and right edges of the second display area 102B are aligned with the left and right edges of the first display area 102A. In this manner, the second display area 102B is horizontally aligned with the first display area 102A. As will be described in more detail below, the spatial alignment of the first display area 102A with the second display area 102B enables certain types of modeless gestures to be made in the second display area 102B relative to corresponding intervals of the first display area 102A.

[0050] exist Figure 1 In the example configuration shown in , the second display area 102B has been divided into a section 110A (the section 110A may be referred to herein as a "virtual touchpad section 110A") and a section 110B (the section 110B may be referred to herein as a "keyboard section 110B"). The virtual touchpad section 110A may be used as a virtual touchpad. For example, Figure 1 As shown, the user can use finger 116A to perform a gesture in the virtual trackpad section 110A. For example, in this example, the user has dragged his finger horizontally in the virtual trackpad section 110A to perform a drag gesture 117. Figure 1The illustrative drag gesture 117 shown in FIG. 1 causes the computing device 100 to move a cursor 118 in the first display area 102A along a path corresponding to the path of the drag gesture 117 .

[0051] The user may also perform a press or tap operation with his finger 116A in the virtual trackpad section 110A to perform an activation operation (e.g., a mouse click) at the location of the cursor 118. Other types of functionality (e.g., multi-touch gestures, such as two-finger or three-finger swipes) may also be performed in the virtual trackpad section 110A.

[0052] exist Figure 1 In the example shown in , the virtual touchpad interval 110A only encompasses a portion of the second display area 102B. In particular, the virtual touchpad 110A has a width equal to the width 108B of the second display area 102B, but has a height 112 that is less than the height 106B of the second display area 102B. In this example, the height 112 of the virtual touchpad interval 110A is selected to provide the functionality disclosed herein while enabling a touch-sensitive virtual keyboard to be presented in the virtual keyboard interval 110B in one configuration. In other configurations, the virtual touchpad interval 110A may have a different height 112. In some configurations, the virtual touchpad interval 110A encompasses the entire width of the second display area 102B and is located in a portion of the second display area 102B adjacent to the first display area 102A. Additionally, in the illustrated example, the top of the keyboard interval 110B (i.e., the top row of keys) is adjacent to and parallel to the bottom edge of the virtual touchpad interval 110A. In other configurations, a physical keyboard replaces the keyboard section 110B of the computing device 100. For example, in some configurations, a physical keyboard can be placed on top of the area typically occupied by a virtual keyboard, providing the tactile feel of physical keys but otherwise providing similar functionality.

[0053] It should be understood that certain relative terms (e.g., height, width, top, bottom, left, right) have been utilized herein to describe Figure 1 In this regard, it should also be understood that these terms have been utilized herein for ease of discussion and that these terms do not limit the configuration of the regions 102A and 102B or the device 100. Other terms may be utilized to describe the display regions 102A and 102B and the spatial relationship between these display regions. Additionally, although Figure 1 The computing device 100 shown in FIG. 1 is illustrated in a landscape orientation, but the device 100 may also be operated in a portrait configuration (ie, by rotating the device 100 ninety degrees). In other configurations, other types of multi-screen computing devices may be utilized.

[0054] Modeless digital pen input to virtual trackpad

[0055] Figures 2 to 5 is a schematic diagram showing aspects of the mechanism disclosed herein for modeless digital pen input to a virtual trackpad. Figures 2 to 5 In the configuration shown in , the virtual trackpad section 110A can receive input from the digital pen 202 without requiring the user to change the operating mode of the virtual trackpad section 110A.

[0056] exist Figure 2 In the configuration illustrated in FIG. 1 , a user may perform a gesture, such as gesture 208 , using finger 116A or multiple fingers in virtual trackpad section 110A of second display area 102B. Figure 2 The illustrative gesture 208 shown in 200 causes the computing device 100 to move the cursor 118 in the first display area 102A along a path corresponding to the path of the gesture 208. The user may also perform a press or tap operation with their finger 116A in the virtual trackpad section 110A to perform an activation operation (e.g., a mouse click) at the location of the cursor 118. Other types of functionality (e.g., multi-touch gestures, such as a two-finger or three-finger swipe) may also be performed in the virtual trackpad section 110A.

[0057] Additionally, the user can write directly in the virtual trackpad section 110A using the digital pen 202 at any time. The user does not need to switch modes to utilize the virtual trackpad functionality described above and utilize the digital pen 202 in the virtual trackpad section 110A. Figure 2 In the example shown in , for example, the user has utilized the digital pen 202 to write digital ink 206A in the virtual touchpad region 110A without changing the input mode.

[0058] When input is received within the virtual trackpad interval 110A, the computing device 100 determines whether the input is a touch input (e.g., input generated by the user's finger 116A) or an input generated by the digital pen 202. To make this determination, the digital pen 202 may be communicatively coupled to the computing device 100 via a suitable interface. Through this connection, the computing device 100 may determine whether the digital pen 202 has been utilized to write in the virtual trackpad interval 110A.

[0059] If the input is a touch input (e.g., a user's finger 116A), the computing device 100 processes the received input as a touch input to the virtual trackpad, for example, to move the cursor 118 or perform an activation operation. If the input is received from the digital pen 202, the computing device 100 processes the input as digital ink. In this way, the user can provide touch input and input from the digital pen 202 to the virtual trackpad section 110A simultaneously without requiring the user to change from a touch input mode to a digital pen mode.

[0060] Processing input received in the virtual touchpad region 110A as digital ink may include, but is not limited to, converting the digital ink into text and providing the text to a program for presenting the text within a field of a UI presented in the first display area 102A. Figure 3 , for example, the user has utilized the digital pen 202 to write digital ink 206A (i.e., “Hello, World”) in the virtual trackpad region 110A. In response, the computing device 100 has converted the digital ink 206A into text and provided the text to an email application executing on the computing device 100. In turn, the email application has presented the recognized text in a field 304A of the UI window 302A that has been selected using the cursor 118. In other configurations, the recognized text may be provided to other types of programs executing on the computing device 100 or another local or remote computing device.

[0061] Processing the input received in the virtual touchpad area 110A as digital ink may also include providing the digital ink to a program for presenting the digital ink within a field of a UI presented in the first display area 102A. Figure 4 , for example, the user has utilized digital pen 202 to write digital ink 206A (i.e., "RubyLu") in virtual trackpad region 110A. In response, computing device 100 has provided digital ink 206B to a program executing on computing device 100 for signing a file. In turn, the program has presented digital ink 206B in field 304B of UI window 302B that has been selected using cursor 118. In other configurations, digital ink 206B may be provided to other types of programs executing on computing device 100 or another local or remote computing device.

[0062] Processing input received in the virtual trackpad region 110A as digital ink may also include, in some configurations, converting the digital ink into text, recognizing a command in the text, and causing the computing device 100 to execute the command. Figure 5In the example illustrated in , for example, the user has utilized digital pen 202 to write digital ink 206C (i.e., "Call 1-(234)-555-1212") in virtual trackpad region 110A. In response, computing device 100 has converted digital ink 206C into text and provided the text to a communication application (not shown). In turn, the communication application has presented the recognized command in UI window 302C and executed the intended command (i.e., call the specified number). In other configurations, the recognized text may be utilized to execute other types of commands on computing device 100 or another local or remote computing device.

[0063] Additionally or alternatively, processing input received in virtual touchpad section 110A as digital ink may include performing other functions. For example, but not limited to, processing input as digital ink may include providing digital ink to a default program for storing notes or digital ink executed on computing device 100 or another computing system, or converting digital ink into text and providing the text to a default program for storing notes executed on computing device 100 or another computing system. Digital ink or text may be provided to such a program in response to receiving digital ink from a user without requiring additional input from the user. In other configurations, computing device 100 may provide digital ink or text recognized from digital ink to other types of programs.

[0064] Figure 6 is a flow chart showing a routine 600, which illustrates Figure 1 The computing device shown in FIG. Figures 2 to 5 It should be understood that the aspects of the operation of the modeless digital pen input on the virtual trackpad shown in the Figure 5 The logical operations described in the and other figures are implemented as (1) a sequence of computer-implemented acts or program modules running on a computing device and / or as (2) interconnected machine logic circuits or circuit modules within a computing device.

[0065] The specific implementation of the technology disclosed herein is a selection problem that depends on the performance and other requirements of the computing device. Therefore, the logical operation described herein is referred to as state, operation, structural device, action, or module in different ways. These states, operations, structural devices, actions, and modules can be realized by hardware, software, firmware, special digital logic, and any combination thereof. It should be understood that more or less operations can be performed compared to the operations shown in the figure and described herein. These operations can also be performed in a sequence different from the sequence described herein.

[0066] The routine 600 begins at operation 602, where the computing device 100 monitors the virtual trackpad interval 110A for input. If input is detected in the virtual trackpad interval 110A, the routine 600 proceeds from operation 604 to operation 606. In operation 606, the computing device 100 determines whether the detected input is a touch input (e.g., input by a user's finger 116A). If the detected input is a touch input, the routine 600 proceeds from operation 606 to operation 610, where the computing device 100 processes the input as a touch input to the virtual trackpad. For example, the computing device 100 may move the cursor 118 or perform an activation operation based on the input. The routine 600 then returns from operation 610 to operation 602 to proceed.

[0067] If the received input is not a touch input, the routine 600 proceeds from operation 606 to operation 608, where the computing device 100 determines whether the received input is input made by the digital pen 202. If the received input is not input made by the digital pen 202, the routine 600 returns from operation 608 to operation 602 to continue. If the input is a digital pen input (i.e., digital ink), the routine 600 proceeds from operation 608 to operation 612, where the computing device 100 processes the received input as digital ink. For example, the computing device 100 may process the received input as digital ink according to the above description with respect to Figures 3 to 5 The routine 600 then proceeds from operation 612 described above to operation 602. In this manner, the computing device 100 can process both touch input and pen input made in the virtual trackpad section 110A without requiring user input to change modes.

[0068] Modeless gesture for summoning the user interface

[0069] Figures 7 to 12 is a schematic diagram illustrating aspects of a mechanism disclosed herein for modelessly summoning display of a transient UI in a virtual trackpad section 110A. In this configuration, the virtual trackpad section 110A can be utilized to initiate presentation of a transient UI in the virtual trackpad section 110A to execute commands and / or view information without requiring the user to change the operating mode of the virtual trackpad section 110A.

[0070] To modelessly provide a transient UI in the virtual trackpad section 110A, the computing device 100 determines whether the received touch input includes a touch gesture that originates outside the edge of the virtual trackpad section 110A of the second display area 102B and ends inside the virtual trackpad section 110B. Figure 7 In the example illustrated in , the user has utilized finger 116A to perform a swipe gesture 706 that originates outside the left edge of the virtual touchpad interval 110A and terminates at a location within the virtual touchpad interval 110A (e.g., an “edge swipe” or a multi-finger gesture that does not involve crossing the edge of the virtual touchpad).

[0071] If the computing device 100 detects a touch gesture that originates outside the edge of the virtual trackpad interval 110A of the second display area 102B and ends inside the virtual trackpad interval 110A, the computing device 100 may display a transient (i.e., temporary) UI in the virtual trackpad interval 110A, such as Figure 7 702. When the transient UI is displayed in the second display area 102B, such as, for example, by "sliding" from an edge of the virtual trackpad interval 110A where the gesture originates into the virtual trackpad interval 110A, the transient UI may be animated.

[0072] In some configurations, the transient UI presented in the virtual trackpad section 110A includes selectable UI elements, such as UI buttons or other types of UI elements. Figure 7 In the example shown in , for example, transient UI 702 includes UI elements 704A and 704B. If computing device 100 determines that a touch input selecting one of the one or more selectable UI elements has been made in virtual trackpad section 110A, computing device 100 may initiate a command corresponding to the selected UI element. For example, a command may be initiated to launch an application that may present a UI in first display area 102A, switch to an already launched application (i.e., switch tasks), or perform another function.

[0073] exist Figure 7 In the example shown in , UI element 704A corresponds to a web browser application, and UI element 704B corresponds to an email application. Figure 8 As shown, in this example, the user has used finger 116A to select UI element 704B. In response to detecting the selection of UI element 704B, the computing device has launched an email application, which in turn has presented UI 802 in first display area 102A. Other types of functions may be performed in response to selection of UI element 704 in transient UI 702 displayed within virtual trackpad zone 110A.

[0074] In some configurations (such as Fig. 9In the configuration illustrated in FIG. 1 ), in response to selection of one of the UI elements, the display of the transient UI 702 is immediately removed from the virtual trackpad area. In other configurations, a touch gesture can be utilized to dismiss the transient UI 702. For example, in Fig.10 and Fig.11 In the example shown in , the user has utilized finger 116A to perform a swipe gesture 1004 that originates outside virtual trackpad section 110A and terminates within virtual trackpad section 110A. As a result, computing system 100 has presented a transient UI 1002 showing a notification in virtual trackpad section 110.

[0075] To dismiss transient UI 1002, the user may utilize finger 116A or multiple fingers to perform a gesture for dismissing transient UI 1002. For example, the user may perform a swipe gesture 1006 from a location inside virtual trackpad section 110A to a location outside virtual trackpad section 110A to dismiss transient UI. Fig.11 Other types of gestures outside of the transient UI 1002, such as a tap gesture, may also be utilized to dismiss the transient UI 1002.

[0076] As is the case when the transient UI is displayed in the virtual touchpad section 110A, the display of the transient UI may also be animated when it is removed from the virtual touchpad section 110A, such as by “sliding” the transient UI outside the virtual touchpad section 110A to the edge of the virtual touchpad section 110A where the gesture terminates.

[0077] In other configurations, an input gesture made from outside the edge of the virtual trackpad interval 110A to a location inside the virtual trackpad interval 110A can perform a function based on the relationship between the end location of the gesture and a UI element presented in the first display area 102A or in the keyboard interval 110B of the second display area 102B. For example, a user can set the audio output volume of the computing device 100 by swiping right from an interval outside the virtual trackpad interval 110A to a location inside the virtual trackpad interval 110A. The position where the gesture ends (e.g., horizontally aligned) relative to the virtual keyboard shown in the keyboard interval 110B specifies the output volume. For example, if the gesture ends at a location that is spatially aligned with the location of the '1' key on the virtual keyboard, the output volume will be set to one (e.g., 10% of the maximum value). If the gesture ends at a location that is spatially aligned (e.g., horizontally aligned) with the location of the '9' key on the virtual keyboard, the output volume will be set to nine (e.g., 90% of the maximum value). In this manner, many different specific edge gestures along a common edge of the virtual trackpad region 110A may be disambiguated by the computing system.Such gestures may also be performed relative to a horizontally aligned physical keyboard.

[0078] Fig.12 is a flow chart showing a routine 1200, which illustrates Figure 1 The computing device 100 shown in FIG. 1 is used to modelessly summon the Figures 7 to 11 Aspects of the operation of the transient UI in the virtual trackpad interval 110A are described. The routine 1200 begins with operation 1202, in which the computing device 100 monitors the virtual trackpad interval 110A for input. If input is detected in the virtual trackpad interval 110A, the routine 1200 proceeds from operation 1204 to operation 1206.

[0079] In operation 1206, the computing device 100 determines whether the detected input is a touch gesture that starts outside the virtual trackpad interval 110A and ends inside the virtual trackpad interval 110A. If the detected input is a touch gesture, the routine 1200 proceeds from operation 1208 to operation 1210, where the computing device displays a transient UI in the virtual trackpad interval 110A. The routine 1200 then returns from operation 1210 to continue at operation 1202.

[0080] If the computing device 100 determines that the received input is not a gesture that originates outside the virtual trackpad interval 110A and ends inside the virtual trackpad interval 110A, the routine 1200 proceeds from operation 1208 to operation 1212. In operation 1212, the computing device 100 determines whether the detected input is a touch gesture for dismissing the transient UI, such as a swipe or tap gesture. If the detected input is a touch gesture for dismissing the transient UI, the routine 1200 proceeds from operation 1212 to operation 1214, in which the computing device removes the transient UI from the virtual trackpad interval 110A. The routine 1200 then returns to operation 1202 from operation 1216 to continue.

[0081] Modeless gestures using relative position

[0082] Figure 13 to Figure 14 is a schematic diagram illustrating aspects of the mechanisms disclosed herein for enabling modeless gestures on a virtual trackpad with functionality determined based on the gesture's origination point relative to an adjacent on-screen UI or physical keyboard. Fig.13 In the example shown in , a first application executing on the computing device 100 presents its UI in a first portion 1302A of the first display area 102A: in this configuration, the left half of the first display area 102A.

[0083] The first portion 1304A of the virtual touchpad section 110A in the second display area 102B is adjacent to and spatially aligned with the first portion 1302A of the first display area 102A. Fig.13 , for example, portion 1302A of first display area 102A has a width 1306A that is the same as width 1308A of portion 1304A of second display area 102B. In this example, first portion 1304A of virtual trackpad zone 110A therefore includes a left half of second display area 102B that is spatially aligned with a left half of first display area 102A that includes a UI generated by a first application (i.e., portion 1302A).

[0084] exist Fig.13 In the example shown in , a second application executed on the computing device 100 presents its UI in a second portion 1302B of the first display area 102A: in this example, the right half of the first display area 102A. A second portion 1304B of the virtual trackpad interval 110A is spatially aligned with the second interval 1302B of the first display area 102A. Fig.13, for example, portion 1302B of the first display area 102A has a width 1306B that is the same as width 1308B of portion 1304B of the second display area 102B. In this example, therefore, second portion 1304B of the virtual trackpad interval 110A includes a right half of the second display area 102B that is spatially aligned with the right half of the first display area 102A (i.e., portion 1302B). In this manner, one application is assigned a portion 1302A of the first display area 102A that corresponds to portion 1304A of the virtual trackpad interval 110A in the second display area 102B that is spatially aligned, and another application is assigned another portion 1302B of the first display area 102A that is spatially aligned with another portion 1304B of the virtual trackpad interval 110A.

[0085] If the computing device 100 receives input in the first portion 1304A of the virtual trackpad interval 110A, the computing device 100 provides the input to the application that is presenting its UI in the adjacent portion 1302A of the first display area 102A. For example, in the illustrated configuration, if the input is received in the left half of the virtual trackpad interval 110A (i.e., portion 1304A), the input is provided to the application presenting its UI in the left half of the first display area 102A (i.e., portion 1302A), or the input is processed by the operating system with reference to the application. The input may be, for example, a multi-touch input gesture (such as Fig.13 ) or input received from a digital pen. For example, these and other types of input can be distinguished from traditional virtual trackpad input for controlling a cursor.

[0086] If the computing device 100 receives input in the second portion 1304B of the virtual trackpad interval 110A, the computing device 100 provides the received input to the application that is presenting its UI in the adjacent portion of the first display area 102A. For example, if the input is received in the right half of the virtual trackpad interval 110A (i.e., portion 1304B), the input is provided to the application that is presenting its UI in the right half of the first display area 102A (i.e., portion 1302B). In this way, user input can be provided to different applications executing on the computing device 100 without requiring the user to change the mode of the virtual trackpad interval 110A. In this regard, it should be understood that although Fig.13 The example shown in includes only two applications, but input can be provided modelessly to more than two applications in a similar manner in other configurations. Input can also be provided to two or more areas presented by a single application, such as an application that features multiple task panes or content areas in its UI.

[0087] Fig.14 Another configuration is illustrated in which computing device 100 can detect a user input gesture that originates in an interval outside virtual trackpad interval 110A and terminates within virtual trackpad interval 110A. In the illustrated example, for example, a vertical swipe gesture using finger 116A originates within keyboard interval 110B and terminates within virtual trackpad interval 110A.

[0088] In response to detecting such a user input gesture, the computing device 100 may execute a command selected based at least in part on the location where the gesture originated. In the illustrated example, for example, the gesture originates on a key of the virtual keyboard and ends in the virtual trackpad interval 110A. In this example, the command to be executed is selected based on the specific key from which the gesture originated. For example, if the gesture originates on the '1' key, the volume of the computing device 100 may be set to 10% of its maximum value. If the gesture originates on the '9' key, the volume may be set to 90% of the maximum value. Other types of commands may be initiated and executed in a similar manner.

[0089] Fig.15 is a flow chart showing a routine 1500, which illustrates Figure 1 Aspects of the operation of computing device 100 shown in FIG. 1 for enabling modeless gestures on a virtual trackpad, the modeless gestures having a control point based on the gesture's starting point relative to the touchpad. Fig.13 and Fig.14 The routine 1500 begins with operation 1502, in which a first application presents its UI in a first portion 1306A of a first display area 102A. The routine 1500 then proceeds to operation 1504, in which a second application presents its UI in a second portion 1306B of the first display area 102A.

[0090] The routine 1500 proceeds from operation 1504 to operation 1506, where the computing device 100 receives input in the virtual trackpad interval 110A of the second display area 102B. The routine 1500 then proceeds to operation 1508, where the computing device 100 determines whether input was made in a portion of the virtual trackpad interval 110A that is adjacent to the portion 1306A of the first display area 102A that contains the UI of the first application. If input was made in the portion of the virtual trackpad interval 110A, the routine 1500 proceeds from operation 1510 to operation 1512, where the input received in operation 1506 is provided to the first application or processed with respect to the first application (e.g., by an operating system). If input was not made in the portion of the virtual trackpad interval 110A, the routine 1500 proceeds from operation 1510 to operation 1514.

[0091] In operation 1514, the computing device 100 determines whether input was made in the portion of the virtual trackpad interval 110A that is adjacent to the portion 1306B of the first display area 102A that contains the UI of the second application. If input was made in the portion of the virtual trackpad interval 110A, the routine 1500 proceeds from operation 1516 to operation 1518, where the input received in operation 1506 is provided to the second application or processed with respect to the second application (e.g., by an operating system). If input was not made in the portion of the virtual trackpad interval 110A, the routine 1500 proceeds from operation 1516 back to operation 1506, where additional input may be received and processed in a similar manner.

[0092] Modeless cursor control and interaction with virtual touch targets

[0093] Figures 16 to 19 is a schematic diagram illustrating aspects of the mechanisms disclosed herein for enabling modeless cursor control and interaction with virtual touch targets. Figures 16 to 19 In the configuration shown in , an area of ​​the display area including a virtual touchpad area 110A on the touch-sensitive display 102B may show touch targets (eg, UI controls) for initiating various types of functionality. Fig.16In the example shown in , for example, the size of the virtual trackpad area 110A has been reduced, and a plurality of UI controls 1602A-1602E have been presented in an area of ​​the second display area 102B surrounding the virtual trackpad area 110A. When the UI controls 1602A-1602E are selected, such as by using a tap gesture in the second display area 102B, the UI controls 1602A-1602E may initiate various types of functionality, such as, for example, launching a program on the computing device 100 or summoning a digital assistant.

[0094] Cursor control may be performed using the virtual trackpad section 110A, and UI controls 1602A to 1602E may be selected without changing the mode of the virtual trackpad. Fig.17 , a touch gesture 1702 for controlling the cursor 118 may originate within the virtual trackpad section 110A and terminate within one of the UI controls 1602A-1602E without causing selection of the UI control where the gesture 1702 ends or without interrupting cursor control. In this manner, if a user's touch gesture inadvertently exits the virtual trackpad section 110A while controlling the cursor 118 and ends within one of the UI controls 1602A-1602E, the UI control will not be selected.

[0095] Similarly, if Fig.18 As shown, a touch gesture 1802 for controlling the cursor 118 may originate in an area surrounding the virtual trackpad (in this example, in UI control 1602B) and end inside the virtual trackpad section 110A or another section of the second display area 102B without causing selection of the UI control from which the gesture originates. In this way, a cursor control gesture, such as gesture 1802, may be performed outside the virtual trackpad section 110A even if it originates from UI control 1602. This may be useful, for example, if the user is not looking at the virtual trackpad section 110A when starting the mouse control gesture.

[0096] The starting position and timing of the user input gesture can be used to disambiguate cursor control and interaction with the UI control without requiring a mode switch and without interrupting cursor control. For example, if a touch gesture is detected within UI control 1602, then if the gesture ends (e.g., the user lifts the finger) within a predetermined time period (e.g., 100ms), the gesture can be considered a selection of UI control 1602. If a touch gesture is not detected within UI control 1602, then the gesture is considered a gesture that begins controlling the position of cursor 118.

[0097] In some configurations, previous gestures may be considered when disambiguating the current gesture. For example, but not limited to, a cursor movement followed by a tap gesture performed near the end location of a previous cursor movement may be interpreted as a selection even if the user's finger is now also over the UI control. In general, a cursor movement followed by a cursor click at the same location or a cursor movement followed by a tap of a UI control at another location on the virtual trackpad may be interpreted as a selection.

[0098] In some configurations, the computing device can detect a user input gesture that originates within the virtual trackpad interval and terminates outside the virtual trackpad interval. In this example, the computing device can execute a command selected based at least in part on the object or location where the gesture terminates.

[0099] In another configuration (in Fig.19 ), a physical input object may be placed on one of several areas of the virtual trackpad region 110A to enable additional functionality, such as, but not limited to, providing direct control of the volume, brightness, and scrolling of the computing device 100. Fig.19 In the example shown in FIG. 1 , for example, the digital dial 1902 (in Fig.19 A circular outline of a portion of the second display area 102B that is represented as a digital dial has been placed in the second display area 102B. The bottom portion of the digital dial 1902 is made of a suitable material for detection by the second display area 102B. In this example, the user has rotated the digital dial 1902 using gesture 1904. In response to such input, the computing device 100 can adjust settings or perform other types of functionality.

[0100] Fig. 20 is a flow chart showing a routine 2000, which illustrates Figure 1 The computing device shown in FIG. Figures 16 to 19 10A . The routine 2000 begins with operation 2002, in which one or more selectable UI controls 1602 are displayed outside or inside the virtual trackpad area 110A. The routine 2000 then proceeds to operation 2004, in which the computing device 100 detects a touch input in the second display area 102B.

[0101] The routine 2000 proceeds from operation 2004 to operation 2006, where the computing device 100 determines whether the touch is located within one of the UI controls 1602. If the touch is located within one of the UI controls 1602, the routine 2000 proceeds from operation 2006 to operation 2008, where the computing device 100 determines whether the touch ends within a predetermined amount of time (e.g., 100 ms). If the touch has ended within the predetermined amount of time, then in operation 2012, the computing device 100 considers the UI control where the touch is located to have been selected. If the touch has not ended within the predetermined amount of time, the routine 2000 proceeds from operation 2010 to operation 2014, where the cursor 118 is moved using a touch gesture.

[0102] In operation 2006, if the computing device determines that the touch detected in operation 2004 is not located in the UI control 1602, the routine 2000 proceeds from operation 2006 to operation 2016. In operation 2016, the computing device 100 determines whether the detected touch is located in the virtual trackpad interval 110A. If the detected touch is located in the virtual trackpad interval 110A, the computing device 100 moves the cursor 118 based on the detected gesture. If the detected touch is not located in the virtual trackpad interval 110A, the routine 2000 returns to operation 2004 to continue, in which additional gestures can be detected and processed in the manner described above.

[0103] Fig.21 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing device that can implement the various techniques presented herein. In particular, Fig.21 The architecture illustrated in FIG. 1 may be utilized to implement the multi-screen computing device 100 described herein. The architecture illustrated may also be utilized to implement other types of computing systems.

[0104] Fig.21The computer 2100 illustrated in the figure includes a central processing unit 2102 ("CPU"), a system memory 2104 (including random access memory 2106 ("RAM") and read-only memory ("ROM") 2108), and a system bus 2110 that couples the memory 2104 to the CPU 2102. A basic input / output system ("BIOS" or "firmware") may be stored in ROM 2018, which contains basic routines that help transfer information between elements within the computer 2100, such as during startup. The computer 2100 also includes a mass storage device 2112 for storing an operating system 2122, application programs, and other types of programs. In various configurations, the functionality described above for enhancing the virtual touchpad is implemented by one or more of these programs. The mass storage device 2112 may also be configured to store other types of programs and data.

[0105] The mass storage device 2112 is connected to the CPU 2102 through a mass storage controller (not shown) connected to the bus 2110. The mass storage device 2112 and its associated computer-readable media provide non-volatile storage for the computer 2100. Although the description of computer-readable media contained herein refers to a mass storage device, such as a hard disk, a CD-ROM drive, a DVD-ROM drive, or a USB memory key, it should be understood by those skilled in the art that the computer-readable media can be any available computer storage media or communication media that can be accessed by the computer 2100.

[0106] Communication media includes computer readable instructions, data structures, program modules, or other data in a modulated data signal (such as a carrier wave or other transport mechanism) and includes any delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics changed or set in a manner that encodes the information in the signal. By way of example and not limitation, communication media include wired media, such as a wired network or a direct wired connection; and wireless media, such as acoustic, radio frequency, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.

[0107] By way of example and not limitation, computer storage media may include volatile and nonvolatile, removable and non-removable media implemented using any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, digital versatile disk ("DVD"), HD-DVD, BLU-RAY or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the computer 2100. For the purposes of the claims, the phrase "computer storage media" and its variations do not by themselves include waves or signals or communication media.

[0108] According to various configurations, the computer 2100 can operate in a networked environment using a logical connection to a remote computer through a network, such as the network 2120. The computer 2100 can be connected to the network 2120 via a network interface unit 2116 connected to the bus 2110. It should be appreciated that the network interface unit 2116 can also be used to connect to other types of networks and remote computer systems. The computer 2100 can also include an input / output controller 2118 for receiving and processing input from a number of other devices, including a keyboard, a mouse, a touch input, a digital pen 202, or a physical sensor, such as a camera. Similarly, the input / output controller 2118 can provide output to one or more display screens, such as the first display area 102A and the second display area 102B.

[0109] As discussed above, the first display area 102A and the second display area 102B are output devices configured to present information in a visual form. In particular, the first display area 102A and the second display area 102B can present graphical user interface ("GUI") elements, text, images, videos, notifications, virtual buttons, virtual keyboards, messaging data, Internet content, device status, time, date, calendar data, preferences, map information, location information, and any other information that can be displayed in a visual form. In some configurations, the first display area 102A and the second display area 102B are liquid crystal displays ("LCDs") utilizing any active or passive matrix technology and any backlight technology (if used). In some configurations, the first display area 102A and the second display area 102B are organic light emitting diode ("OLED") displays. Other display types are contemplated.

[0110] As discussed above, the second display area 102B can be a touch-sensitive display configured to detect the presence and location of a touch. Such a display can be a resistive touch screen, a capacitive touch screen, a surface acoustic wave touch screen, an infrared touch screen, an optical imaging touch screen, a dispersive signal touch screen, an acoustic pulse recognition touch screen, or can utilize any other touch screen technology. In some configurations, the touch screen is included on top of the display in the form of a transparent layer to enable a user to interact with objects or other information presented on the display using one or more touches.

[0111] The second display area 102B can be configured to detect multiple touches simultaneously. In some configurations, the second display area 102B is configured to detect discrete touches, single-touch gestures, and / or multi-touch gestures. For convenience, these touch gestures are collectively referred to as "gestures" herein. Several gestures will now be described. It should be understood that these gestures are illustrative and are not intended to limit the scope of the appended claims. Moreover, the described gestures, additional gestures, and / or alternative gestures can be implemented in software used with the second display area 102B. Thus, developers can create gestures specific to specific applications.

[0112] In some configurations, the second display area 102B supports a tap gesture, wherein the user taps the second display area 102B once on the items presented in the second display area 102B. In some configurations, the second display area 102B supports a double-click gesture, wherein the user taps the second display area 102B twice on the items presented in the second display area 102B. For various reasons, the double-click gesture can be used, including but not limited to, zooming in or out in stages. In some configurations, the second display area 102B supports a tap and hold gesture, wherein the user taps the second display area 102B and maintains contact for at least a predefined time. For various reasons, the tap and hold gesture can be used, including but not limited to, opening a context-specific menu.

[0113] In some configurations, the second display area 102B supports a pan gesture in which a user places a finger in the second display area 102B and maintains contact with the second display area 102B while moving the finger in the second display area 102B. The pan gesture may be used for a variety of reasons, including, but not limited to, moving through a screen, image, or menu at a controlled rate. Multiple finger pan gestures are also contemplated.

[0114] In some configurations, the second display area 102B supports a flick gesture, in which a user swipes a finger in the direction the user wants the screen to move. The flick gesture can be used for various reasons, including, but not limited to, scrolling through menus or pages horizontally or vertically. In some configurations, the second display area 102B supports a pinch and stretch gesture, in which a user pinches two fingers (e.g., a thumb and index finger) in the second display area 102B or spreads the two fingers apart. The pinch and stretch gesture can be used for various reasons, including, but not limited to, gradually zooming in or out of a website, map, or picture.

[0115] Although the gestures described above have been presented with reference to the use of one or more fingers for performing the gestures, other appendages (such as a digital pen) may also be used to interact with the second display area 102B. Thus, the above gestures should be understood to be illustrative and should not be construed as limiting in any way.

[0116] It should be understood that the software components described herein, when loaded into the CPU 2102 and executed, the CPU 2102 and the entire computer 2100 can be transformed from a general-purpose computing device into a special-purpose computing device customized to support the functionality presented herein. The CPU 2102 can be composed of any number of transistors or other discrete circuit elements, which can assume any number of states individually or collectively. More specifically, in response to the executable instructions contained in the software modules disclosed herein, the CPU 2102 can operate as a finite state machine. These computer executable instructions can transform the CPU 2102 in the following manner: specify how the CPU 2102 transforms between states, thereby transforming the transistors or other discrete hardware elements that constitute the CPU 2102.

[0117] Encoding the software modules presented herein may also transform the physical structure of the computer-readable medium presented herein. In different implementations of this specification, the specific transformation of the physical structure depends on various factors. Examples of such factors include, but are not limited to, the technology used to implement the computer-readable medium, whether the computer-readable medium is characterized as a primary storage device or an auxiliary storage device, etc. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed herein may be encoded on the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software may transform the state of transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software may also transform the physical state of such a component so that data is stored thereon.

[0118] As another example, the computer readable medium disclosed herein can be implemented using magnetic or optical technology. In such an implementation, when the software is encoded in a magnetic or optical medium, the software presented herein can transform the physical state of the magnetic or optical medium. These transformations can include: changing the magnetic properties of specific locations within a given magnetic medium. These transformations can also include: changing the physical characteristics or properties of specific locations within a given optical medium to change the optical properties of those locations. Other transformations of the physical medium are possible without departing from the scope and spirit of this specification, wherein the foregoing examples are provided only to facilitate this discussion.

[0119] In view of the above, it should be understood that many types of physical transformations occur in the computer 2100 in order to store and execute the software components presented herein. It should also be understood that the software components presented herein may be used. Fig.21 The architecture for computer 2100 shown in FIG. 1 or a similar architecture may be used to implement other types of computing devices, including handheld computers, video game devices, embedded computer systems, mobile devices (such as smart phones, tablets, and AR / VR devices), and other types of computing devices known to those skilled in the art. It is also contemplated that computer 2100 may not include Fig.21 All of the components shown in the figure may include Fig. 20 Other components not explicitly shown may be used in conjunction with Fig.21 A completely different architecture than the one shown in .

[0120] It should be understood that for ease of discussion, the Fig.21 It should also be understood that the illustrated computing architecture may include and utilize many more computing components, devices, software programs, network devices, and other components not specifically described herein.

[0121] The disclosure presented herein also covers the subject matter set forth in the following clauses:

[0122] Item 1: A computing device comprising: a processor; a touch-sensitive display device providing a first display area; and a memory storing instructions executable by the processor to: determine whether input has been received within a virtual touchpad interval of the first display area; in response to determining that the input has been received within the virtual touchpad interval, determine whether the input includes touch input or input received from a digital pen; if the input received within the virtual touchpad interval includes touch input, process the input as touch input to the virtual touchpad; and if the input received within the virtual touchpad interval includes input received from a digital pen, process the input as digital ink.

[0123] Clause 2: A computing device according to clause 1, wherein the computing device includes a second display area, and wherein processing the input as digital ink includes: converting the digital ink into text; and providing the text to a program or an operating system for presentation of the text within a field of a user interface (UI) presented in the second display area.

[0124] Clause 3: A computing device according to any of clauses 1 to 2, wherein the computing device includes a second display area, and wherein processing the input as digital ink includes: providing the digital ink to the program for presentation of the digital ink within a field of a user interface (UI) presented in the second display area.

[0125] Clause 4: The computing device of any of clauses 1 to 3, wherein processing the input as digital ink comprises: converting the digital ink into text; recognizing commands in the text; and causing the computing device to execute the commands.

[0126] Clause 5: The computing device of any of clauses 1 to 4, wherein processing the input as digital ink comprises providing the digital ink to a default program or operating system feature for storing notes.

[0127] Clause 6: The computing device of any of clauses 1 to 5, wherein processing the input as digital ink comprises: converting the digital ink into text; and providing the text to a default program for storing notes.

[0128] Item 7: A computing device comprising: a processor; a touch-sensitive display device providing a display area; and a memory storing instructions executable by the processor to: determine whether a touch input has been received within a virtual touchpad interval of the display area; in response to determining that the input has been received within the virtual touchpad interval, determine whether the touch input includes a touch gesture that originates outside an edge of the virtual touchpad interval of the display area and terminates within the virtual touchpad interval; and in response to detecting the touch gesture, display a transient user interface (UI) within the virtual touchpad interval.

[0129] Clause 8: The computing device of clause 7, wherein the transient UI is animated while the transient UI is displayed within the virtual trackpad interval.

[0130] Clause 9: A computing device according to any of clauses 7 to 8, wherein the transient UI has one or more selectable UI elements, and wherein the memory stores other instructions executable by the processor to: determine whether a touch input selecting one of the one or more selectable UI elements has been performed in a virtual touchpad interval; and in response to determining that a touch input selecting one of the one or more selectable UI elements has been performed in the virtual touchpad interval, initiating a command corresponding to the selected one of the one or more selectable UI elements, and removing the display of the transient UI from the virtual touchpad interval.

[0131] Clause 10: The computing device of clauses 7 to 9, wherein the computing device includes a second display area, and wherein initiation of the command causes an application or an operating system executing on the computing device to display the UI in the second display area.

[0132] Clause 11: A computing device according to any of clauses 7 to 10, wherein the memory stores other instructions executable by the processor to: in response to determining that the input has been received within the virtual touchpad interval, determine whether the touch input includes a touch gesture that originates within the virtual touchpad interval and terminates outside the edge of the virtual touchpad interval; and in response to detecting that the touch gesture includes a touch gesture that originates within the virtual touchpad interval and terminates outside the edge of the virtual touchpad interval, remove the display of the transient UI from the virtual touchpad interval.

[0133] Clause 12: The computing device of any of clauses 7 to 11, wherein upon removing display of the transient UI from the virtual trackpad interval, the transient UI is animated.

[0134] Item 13: A computing device comprising: a processor; a first display device providing a first display area; a touch-sensitive display device providing a second display area; and a memory storing instructions executable by the processor to: display a UI control in the second display area, which UI control will initiate a corresponding function on the computing device when selected; receive a first user input gesture, which first user input gesture originates in a virtual touchpad area of ​​the second display area and terminates at the UI control; and in response to receiving the first user input gesture, move a cursor in the first display area without activating the UI control.

[0135] Clause 14: A computing device according to clause 13, wherein the memory stores other instructions executable by the processor to: receive a second user input gesture, the second user input gesture originating within the UI control; determine whether the second user input gesture will select the UI control; and in response to determining that the second user input gesture will not select the UI control, move the cursor in the first display area without activating the UI control.

[0136] Clause 15: A computing device according to any one of clauses 13 to 14, wherein determining whether the second user input gesture is to select a UI control comprises: if the second user input gesture comprises a press on the UI control that is not lifted from the UI control within a predetermined amount of time, then determining that the second user input gesture will not select the UI control.

[0137] Clause 16: A computing device according to any of clauses 13 to 15, wherein the memory stores other instructions executable by the processor to: receive a second user input gesture after the first user input gesture; and perform a function in response to receiving the second user input gesture, the function being selected at least in part based on the first user input gesture.

[0138] Clause 17: A computing device according to any of clauses 13 to 16, wherein the memory stores other instructions executable by the processor to: detect the presence of a physical input object in the second display area; receive input from the physical input object; and perform a function on the computing device based on the input received from the physical input object.

[0139] Clause 18: The computing device of any of clauses 13 to 17, wherein the physical input object comprises a number dial.

[0140] Clause 19: A computing device according to any of clauses 13 to 18, wherein the function comprises: changing a setting of the computing device.

[0141] Clause 20: The computing device of any of clauses 13 to 19, wherein the setting comprises: controlling one or more of a volume or brightness of the computing device.

[0142] Based on the foregoing, it should be appreciated that techniques for modelessly enhancing the functionality provided by a virtual trackpad have been disclosed herein. Although the subject matter presented herein has been described using language specific to computer structural features, methods and transformational actions, specific computing machines, and computer-readable media, it should be understood that the subject matter set forth in the appended claims is not necessarily limited to the specific features, actions, or media described herein. Rather, the specific features, actions, and media are disclosed as example forms of implementing the claimed subject matter.

[0143] The subject matter described above is provided by way of illustration only and should not be construed as limiting. Various modifications and changes may be made to the subject matter described herein without following the illustrated and described example configurations and applications and without departing from the scope of the present disclosure as set forth in the following claims.

Claims

1. A computing device, comprising: processor; A touch-sensitive display device provides a first display area; a second display device separate from the touch-sensitive display device, wherein the second display device is aligned with the touch-sensitive display device and is configured to display content generated from input received at the touch-sensitive display device; as well as A memory storing instructions executable by the processor to: determining whether input has been received within a virtual touchpad interval of the first display area; In response to determining that input has been received within the virtual touchpad interval, determining whether the input includes touch input or input received from a digital pen; If the input received within the virtual touchpad interval includes a touch input, processing the input as a touch input to the virtual touchpad; as well as If the input received within the virtual touchpad interval includes input received from a digital pen, the input is processed as digital ink for display in an input field displayed in a user interface displayed in the second display device, while concurrently controlling cursor movement based on the touch input, wherein processing the input as digital ink includes providing the digital ink to a program function or operating system feature for storing data.

2. The computing device of claim 1, wherein the computing device includes a second display area, and wherein processing the input as digital ink comprises: converting the digital ink into text; as well as The text is provided to a program or an operating system for presentation of the text within a field of a user interface presented in the second display area.

3. The computing device of claim 1 , wherein the computing device includes a second display area, and wherein processing the input as digital ink comprises: The digital ink is provided to a program for presentation of the digital ink within a field of a user interface presented in the second display area.

4. The computing device of claim 1 , wherein processing the input as digital ink comprises: converting the digital ink into text; identifying commands in the text; as well as The computing device is caused to execute the command.

5. The computing device of claim 1 , wherein processing the input as digital ink comprises: converting the digital ink into text; as well as The text is provided to a default program for storing sticky notes.

6. A computing device comprising: processor; A first display device, providing a first display area; a touch-sensitive display device separate from the first display device, providing a second display area; as well as A memory storing instructions executable by the processor to: determining whether a touch input has been received within a virtual touchpad area of ​​the second display area; In response to determining that input has been received within the virtual touchpad interval, determining whether the touch input includes a touch gesture that originates outside an edge of the virtual touchpad interval of the second display area and terminates within the virtual touchpad interval; in response to detecting the touch gesture originating outside the edge of the virtual trackpad interval of the second display area and terminating inside the virtual trackpad interval, displaying a transient user interface within the virtual trackpad interval, wherein the transient user interface remains displayed until subsequent user input is detected at the transient user interface, the subsequent user input indicating a selection of at least one of a plurality of elements displayed in the transient user interface; as well as In response to the user input indicating the selection of the at least one element in the transient user interface, the at least one element is displayed in response to the touch gesture originating outside the edge of the virtual touchpad interval of the second display area and terminating inside the virtual touchpad interval, displaying an application user interface in the first display area in the first display device or activating a program function or operating system feature for storing data. 7 . The computing device of claim 6 , wherein the transient user interface is animated while the transient user interface is displayed within the virtual trackpad interval.

8. The computing device of claim 6, wherein the transient user interface has one or more selectable user interface elements, and wherein the memory stores other instructions executable by the processor to: determining whether a touch input selecting one of the one or more selectable user interface elements has been performed in the virtual trackpad interval; and In response to determining that a touch input selecting one of the one or more selectable user interface elements has been performed in the virtual trackpad interval, initiating a command corresponding to the selected one of the one or more selectable user interface elements, and The display of the transient user interface is removed from the virtual trackpad interval.

9. The computing device of claim 8, wherein the computing device includes a second display area, and wherein initiation of the command causes an application or an operating system executing on the computing device to display a user interface in the second display area.

10. The computing device of claim 6, wherein the memory stores other instructions executable by the processor to: In response to determining that input has been received within the virtual touchpad interval, determining whether the touch input includes a touch gesture that originates within the virtual touchpad interval and terminates outside the edge of the virtual touchpad interval; and Responsive to detecting that the touch gestures include a touch gesture that originates within the virtual touchpad interval and terminates outside the edge of the virtual touchpad interval, removing the display of the transient user interface from the virtual touchpad interval.

11. A computing device for providing cursor control while mitigating inadvertent selection of user interface controls, comprising: processor; A first display device, providing a first display area; A touch-sensitive display device providing a second display area; as well as A memory storing instructions executable by the processor to: displaying the user interface control in the second display area, the user interface control initiating a corresponding function on the computing device when selected, wherein the user interface control is located outside the virtual trackpad interval; receiving a first user input gesture, the first user input gesture originating in the virtual trackpad interval of the second display area and terminating at the user interface control in the second display area and outside the virtual trackpad interval; as well as In response to receiving the first user input gesture at the touch-sensitive display device that originates in the virtual touchpad interval of the second display area and terminates at the user interface control in the second display area and outside the virtual touchpad interval, moving a cursor in the first display area of ​​the first display device while limiting activation of functionality associated with the user interface control, wherein selection of the user interface control activates a program function or operating system feature for storing data.

12. The computing device of claim 11, wherein the memory stores other instructions executable by the processor to: receiving a second user input gesture, the second user input gesture originating from within the user interface control; determining whether the second user input gesture will select the user interface control; as well as In response to determining that the second user input gesture will not select the user interface control, moving the cursor in the first display area without activating the user interface control.

13. The computing device of claim 12, wherein determining whether the second user input gesture is to select the user interface control comprises: If the second user input gesture comprises a press on the user interface control that is not lifted from the user interface control within a predetermined amount of time, it is determined that the second user input gesture will not select the user interface control. The computing device of claim 12 , wherein the user input terminates within the virtual trackpad interval.

15. The computing device of claim 11, wherein the memory stores other instructions executable by the processor to: receiving a second user input gesture subsequent to the first user input gesture; and In response to receiving the second user input gesture, a function is executed, the function being selected based at least in part on the first user input gesture.

16. The computing device of claim 11, wherein the memory stores other instructions executable by the processor to: detecting the presence of a physical input object in the second display area; receiving input from the physical input object; and A function is performed on the computing device based on the input received from the physical input object.

17. The computing device of claim 16, wherein the physical input object comprises a number dial.

18. The computing device of claim 16, wherein the function comprises changing a setting of the computing device, the setting comprising one or more of: controlling a volume of the computing device, or controlling a volume brightness of the computing device.

19. The computing device of claim 11, wherein the user interface control is located outside the virtual trackpad interval.

Images