Predictive gesture optimization for moving objects across display boundaries

Abstract

Disclosed herein is a technology that enables a foldable computing device with multiple screen regions to perform inter-region user interface (UI) operations in response to intra-region UI gestures. For example, a UI gesture that starts and ends in a first region can be used to move a window from the first region to a second region. The disclosed technology solves the above-mentioned technical problems by providing concise and accurate UI gestures that enable a foldable computing device to perform inter-region UI operations. The disclosed technology further solves the above-mentioned technical problems by combining different types of UI gestures to increase the gesture accuracy and expressiveness of inter-region UI operations.

Description

Background Art

[0001] Foldable computing devices include devices with two screens connected by a hinge or devices with a bendable screen. These types of devices can provide advantages over traditional computing devices, such as laptops. However, typically, these devices implement user interface ("UI") paradigms that were originally designed for computing devices with traditional form factors. Therefore, the UI provided by the foldable computing device can be cumbersome and prone to errors, which can lead to incorrect or unintentional user input and unnecessary consumption of computing resources.

[0002] It is in response to these and other technical challenges that the disclosure presented herein is addressed. Summary of the invention

[0003] Disclosed herein is a technology that enables a foldable computing device with multiple screen regions to perform inter-region UI operations in response to an intra-region UI gesture. For example, a UI gesture that starts and ends in a first region can be used to move a window from the first region to a second region. The disclosed technology solves the above technical problems by providing concise and accurate UI gestures that enable a foldable computing device to perform inter-region UI operations.

[0004] By combining different types of UI gestures to increase the gesture accuracy and expressiveness of inter-region UI operations, the disclosed technology further solves the above-mentioned technical problems. Through the implementation of the disclosed technology, the UI can be provided by a foldable device that is easier to use and causes fewer user input errors. In addition, by avoiding processes associated with inefficient navigation of the UI and unintentional or incorrect user input, the utilization of computing resources by the foldable computing device can be reduced. Other technical benefits not specifically mentioned herein can also be achieved by implementing the disclosed subject matter.

[0005] In one embodiment, a foldable computing device is configured to receive a UI gesture that begins and ends in a first display area. In response, the foldable computing device may perform a UI operation that is at least partially displayed in a second display area. Details of such a foldable computing device are described below with respect to Figures 1A to 12 supply.

[0006] Also disclosed herein is a foldable computing device configured to receive a combination of different types of UI gestures and, in response, perform operations utilizing multiple display areas. For example, a drag-and-drop gesture may be combined with a flick gesture such that certain aspects of the drag-and-drop gesture modify the operations caused by the flick gesture. Details regarding such a foldable computing device are provided below with respect to FIGS. 13 to 1.

[0007] It should also be understood that the above subject matter can be implemented as a computer-controlled apparatus, a computer-implemented method, a computing device, or as an article such as a computer-readable medium. These and various other features will be clear by reading the following "Detailed Description" and viewing the associated drawings.

[0008] This Summary is provided to present a brief description of some aspects of the disclosed technology in a simplified form, which aspects are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended that this Summary be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A and 1B is a device posture diagram illustrating configuration and operational aspects of an articulated computing device implementing the disclosed technology in one particular configuration;

[0010] FIG. 2A to FIG. 2C is a device posture diagram illustrating aspects of the configuration and operation of a flexible computing device implementing the disclosed technology in one particular configuration;

[0011] FIG. 3A to FIG. 3F is a diagram of a device user interface illustrating aspects of the prior art for moving a window from one area to another;

[0012] 4A to 4D is a device user interface diagram showing aspects of a symbolic “flick” gesture originating in a first display area, crossing a dynamically determined threshold in the first display area, and causing a window to move to a second display area, according to one embodiment;

[0013] FIG. 5A to FIG. 5F is a device user interface diagram illustrating aspects of a manipulation gesture originating over a file icon in a first display area, entering a dynamically determined target range, and causing an operation to be performed on a corresponding file in a second display area, according to one embodiment;

[0014] Figure 6 is a device user interface diagram showing aspects of a target range that is dynamically determined when a file icon is dragged over a potential destination in a first area, according to one embodiment;

[0015] Figure 7 is a device user interface diagram showing aspects of a target range that is dynamically determined as a potential target exits along an expected gesture path, according to one embodiment;

[0016] Figure 8is a device user interface diagram showing aspects of prediction of actions that a user may perform based in part on gesture extrapolation ("extrapolation") according to one embodiment;

[0017] Fig. 9 is a device user interface diagram showing aspects of a prediction of an operation that a user intends to perform based in part on an analysis of a gesture, according to one embodiment;

[0018] Fig.10 is a device user interface diagram of a description of aspects showing an amount of confidence in a prediction of an action that a user may perform, according to one embodiment;

[0019] FIG. 11A to FIG. 11D is a device user interface diagram illustrating aspects of a symbolic “flick” gesture according to one embodiment;

[0020] Fig.12 is a flow chart of a routine illustrating aspects of the operation of a predictive gesture optimized foldable computing device for moving an object across a display boundary, according to one embodiment;

[0021] Fig.13A and Fig. 13B is a device user interface diagram showing aspects of a combination of a manipulation gesture and a symbol gesture for opening a document according to one embodiment;

[0022] Fig.14A and Fig. 14B is a device user interface diagram showing aspects of a combination of manipulation gestures and symbol gestures for opening documents on different display areas according to one embodiment;

[0023] Fig.15 is a flow chart illustrating a routine for composite symbols and manipulating aspects of a gesture language for multi-screen windowing ("windowing"), according to one embodiment; and

[0024] Fig.16 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing device that may implement aspects of the techniques presented herein. DETAILED DESCRIPTION

[0025] The following detailed description relates to techniques for predictive gesture optimization for moving objects across display boundaries, and composite symbols and manipulation gesture languages ​​for multi-screen windowing. As briefly discussed above, implementation of the disclosed technology can enable the UI to provide easier use and result in fewer user input errors. Therefore, compared to previous solutions, the use of computing resources can be reduced by avoiding processing associated with inefficient navigation of the UI and unintentional or incorrect user input. Other technical benefits not specifically mentioned herein can also be achieved by implementing the disclosed subject matter.

[0026] Those skilled in the art will recognize that the subject matter disclosed herein can be implemented with various types of computing systems and modules, at least some of which are described in detail below. Those skilled in the art will also understand that the subject matter described herein can be practiced with other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, computing or processing systems embedded in devices (such as wearable devices, automobiles, home automation, etc.), computing or processing systems embedded in devices (such as wearable computing devices, automobiles, home automation, etc.), etc.

[0027] In the following detailed description, reference is made to the accompanying drawings that form a part thereof and are shown by way of illustration of specific configurations or examples. Referring now to the drawings, in which like numerals represent like elements throughout the several figures, aspects of various techniques for providing inter-region UI operations in response to intra-region UI gestures and for combining different types of UI gestures to perform inter-region operations will be described.

[0028] Before discussing certain aspects of the disclosed technology, a brief introduction to foldable computing devices (which may be referred to herein as "foldable devices") will be provided. As briefly discussed above, foldable devices include multi-screen form factor devices having two physical display screens joined together using a hinge or other equivalent mechanism (which may be referred to herein as "articulated devices"). By manipulating the orientation of the display screens relative to each other in a hinged manner, such devices can be configured into a variety of postures, some of which are described below with respect to Figure 1A and Figure 1B Describe in more detail.

[0029] Foldable devices also include computing devices with bendable displays (which may be referred to herein as "bendable devices"), such as computing devices that utilize flexible screen technology. When such a device is not bent, it presents a single display surface. When bent, these devices present a single display surface with a crease in the middle. Bendable devices can also be configured into a variety of postures by varying the amount of bending, some of which will also be referenced below. Figure 1A and Figure 1BDescribe in more detail.

[0030] The display screens of foldable computing devices may be touch-sensitive, thereby enabling such devices to recognize touch or stylus input, presses, swipes, and other types of gestures, some of which are described below. Of course, these devices may also be used while being held in a variety of orientations, some of which are described below with respect to Figure 1A and Figure 1B Give a description.

[0031] Reference now Figure 1A and 1B , details about multiple postures of a foldable display device will be described. Figure 1A and Figure 1B The illustrated examples show some possible postures of the articulated device 102. However, it should be understood that the illustrated postures are generally applicable to bendable devices as well.

[0032] like Figure 1A As shown, by changing the angle of the hinge 108 connecting the two display devices 104A and 104B (which may be referred to as "displays" in this article), the hinged device 102 can be configured into a variety of postures. Figure 1A , for example, the hinged device 102 is shown in: a flat, unbent posture, wherein the display screens 104A and 104B are parallel to each other in a portrait orientation; a fully opened, fully bent posture, wherein a single display 104B faces one direction in a portrait orientation and the other display 104A faces the opposite direction; and a partially opened posture in a portrait orientation, wherein the hinge 108 is bent. The hinged device 102 may also be configured in a closed posture (not shown) in which neither display screen is visible.

[0033] exist Figure 1B , the hinged device 102 is shown in: a flat posture, in which the display screens 104A and 104B are in an unbent landscape orientation; a tented posture, in which the hinged device 102 stands on an edge opposite the hinge 108 and presents the displays 104A and 104B in opposing orientations; a supported posture, in which the angle of the hinge 108 enables the hinged device 102 to stand on the display 104B to present a single display 104A; and a partially opened posture, in which one display 104A is laid flat and the other display 104B is at least partially upright, thereby forming a configuration similar to a conventional laptop computer. At this point, it should be understood that Figure 1A and Figure 1B The postures shown are illustrative and other postures are possible.

[0034] Also like Figure 1A and Figure 1BAs shown, display device 104A provides a first display area 106A including all or part of display 104A. Similarly, display device 104B provides a second display area 106B including all or part of display 104B. However, the disclosed embodiments are equally applicable to any display area, including a display area as one of a plurality of areas within a display device and / or a display area across a display device. The first display area 106A and the second display area 106B can present graphical user interface ("GUI") elements, text, images, videos, notifications, virtual buttons, virtual keyboards, messaging data, Internet content, device status, time, date, calendar data, preferences, map information, location information, and any other information that can be presented in a visual form. The various uses of the display areas 106A and 106B provided by the articulated device 102 will be described below.

[0035] Reference now Figure 2A and Figure 2B , details about the configuration and operation of the bendable device 202 will be provided. As described above, the bendable device 202 can also be configured as described above with respect to Figure 1A and Figure 1B The posture described is the same or similar posture. For example, in Figure 2B In the example posture shown, bendable device 202 has been placed in a flat, unbent position, thereby presenting the entire display 104C. Bendable device 202 may also be configured in a closed posture in which no display area is visible.

[0036] exist Figure 2B and Figure 2C In the example posture shown, the bendable device 202 has been partially bent, similar to Figure 1A and Figure 1B Partially opened position shown. In particular, Figure 2B In the example shown, bendable device 202 has been bent when screen 104C is in landscape orientation. Figure 2C In the example shown, device 202 has been bent and screen 104C is in a portrait orientation.

[0037] When the bendable device 202 is bent, a crease or "fold" 204 is formed in the display 104C. As used herein, the term "fold" may refer to the area of ​​the foldable device where it is folded (i.e., the area of ​​the hinge 108 on the hinged device 102 or the area of ​​the display of the bendable device 202 where it is bent).

[0038] As with the articulated device 102, the bendable device 202 may also provide one or more display regions. However, in the case of the bendable device 202, the number of available display regions may vary based on the posture of the device. For example, when the bendable device 202 is in a position such as Figure 2A When the bendable device 202 is in a bent position, such as Figure 2B As shown, two display areas 106D and 106E may be provided on the display 104C. Various details regarding the configuration and use of the display area 106 (rendered using dashed lines throughout the figure) provided by the foldable device are provided below.

[0039] Predictive gesture optimization for moving objects across display boundaries

[0040] Reference now FIG. 3A to FIG. 3F , details about the prior art inter-region gestures will be provided. It can be understood that FIG. 3A to FIG. 3F (and 4A to 11D ) is shown in two dimensions for ease of illustration. However, this is not intended to indicate that the foldable device 301 can be laid flat. 4A to 4D and Figure 6 In the embodiment, the foldable device 301 is in a bent posture, such as Figure 1B shown.

[0041] In discussion FIG. 3A to FIG. 3F Before and other figures, it should also be understood that certain relative terms (e.g., height, width, top, bottom, left, right) can be used in this article to describe the configuration of the display and display area described in this article. In this regard, it should also be understood that these terms have been used in this article to facilitate discussion, rather than to limit the configuration of the display area or UI elements. Other terms can be used to describe the display, display area, UI elements, and their spatial relationship to each other. It should also be understood that, although usually described separately, the various embodiments briefly described above and described in further detail below can be used in combination with each other.

[0042] Figures 3A to 3F FIG. 3 is a device user interface diagram 300 illustrating aspects of the prior art for moving a window from one area to another. Specifically, FIG. 3A to FIG. 3F An inter-region "drag and drop" gesture 330 is shown starting in region 106B and ending in region 106A. Originally designed for tablet form factors such as mobile phones and tablet computers, these gestures are cumbersome and error-prone when applied to integrated multi-screen devices like foldable device 301.

[0043] For example, a drag-and-drop gesture 330 that spans two regions may have an ambiguous meaning after the input device (e.g., finger 116) leaves region 106B, but before it enters region 106A. This is because hinge 108 may not be touch-sensitive, and therefore foldable device 301 may not receive data indicating that gesture 330 will continue into region 106A, e.g., moving a window from region 106B to region 106A, or if gesture 330 has ended at the edge of region 106B, e.g., docking a window to the edge of region 106B. To handle this ambiguity, foldable device 301 may pause to determine whether a second gesture in region 106A is received and whether it is a continuation of the first gesture. This pause can reduce the responsiveness of the user interface when there is no second gesture, resulting in a poor user experience.

[0044] Another challenge with inter-region gestures may occur when the foldable device 301 is in a posture that makes inter-region gestures awkward, error-prone, or impossible, such as FIG. 3A to FIG. 3F For example, when foldable device 301 is in a tent shape or oriented as a traditional laptop computer, it may be difficult for a user to continuously and accurately perform gestures that span regions 106B and 106A. When foldable device 301 is propped up, it may be difficult to perform gestures that span regions 106B and 106A if display region 106B is difficult to access.

[0045] Figure 3A The beginning of inter-region gesture 330 is shown. Touch 304 on title bar 306 of window 308 begins gesture 330. Touch 304 may be a press that is maintained throughout at least the entire portion of gesture 330 performed in region 106B. Figure 3A Touch 304 is depicted as being caused by a finger, although the gesture may also be caused by a stylus, pen, or other physical object capable of activating a touch screen sensor of region 106B.

[0046] Figure 3B Shown in Figure 3A 3. Pointer 302 continues to press while moving along path 310 toward location 312 on hinge 108. Window 308 may move with pointer 302 to create the effect that it is being dragged by pointer 302. Pointer 302 may be a finger, stylus, pen, or any physical object capable of activating the touch sensors of display areas 106A and 106B.

[0047] Figure 3CPointer 302 is shown sliding out of region 106B onto hinge 108. In this example, hinge 108 is not touch sensitive, and therefore sliding off region 106B ends the gesture along path 310. Pointer 302 may continue to press location 314 of hinge 108 as part of a continuous sliding motion, even though the press is no longer registered.

[0048] Figure 3D Pointer 302 is shown continuing to slide across hinge 108 to position 318. During this time, no input is received by foldable device 301. As a result, window 308 remains in the position at which pointer 302 ceased to register as input to region 106B.

[0049] Figure 3E Pointer 302 is shown after having pressed region 106A at position 320 and continuing to slide upward along path 322 to position 324. Foldable device 301 can interpret this motion as a continuation of the gesture along path 310 based on factors such as proximity in time, proximity in space, continuity in the directions of paths 310 and 322, etc. However, associating the two paths as part of gesture 330 is error prone.

[0050] Figure 3F Pointer 302 is shown completing the drag and drop operation of window 308 by continuing to slide upward along path 322 to position 328. As pointer 302 slides along path 322, foldable device 301 can synchronously move window 308, i.e., can move window 308 so that the same portion of title bar 306 remains under pointer 302 throughout the drag operation. Once pointer 302 is lifted from region 106A at position 328, drag 330 is completed and a drop operation is performed to leave window 308 at the new location.

[0051] 4A to 4D4 is a device user interface diagram 400, which shows aspects of a symbolic "flick" gesture 414 originating from region 106B and crossing a threshold 410 dynamically determined in region 106B. In one configuration, a flick gesture is a gesture that generally moves in one direction to allow for a certain curvature as an error margin. Flick gestures can be used for a variety of reasons, including but not limited to flipping through documents horizontally or vertically, moving windows, icons, or other UI elements to different areas, moving icons to folders, adjusting window sizes (e.g., maximizing or minimizing windows), etc. In some configurations, a flick gesture can be interpreted as being similar to a keyboard shortcut-a defined gesture that, once recognized by the system, causes a command to be executed. Threshold 410 is dynamically calculated because the position and orientation of threshold 410 depends on many factors, some of which can be updated as flick 414 proceeds. As pointer 302 approaches threshold 410, threshold 410 may be visible or become visible. Additionally or alternatively, the foldable device 301 may generate tactile, audio, and / or visual feedback when the pointer 302 crosses the threshold 410 .

[0052] In some configurations, a symbolic gesture may be initiated on a UI item, for example, based on touching, pressing, hovering, etc. of a UI item. UI items may include windows, dialog boxes, icons, menus, application content, etc. In other embodiments, a context-independent symbolic gesture may be initiated on a desktop background, independent of a UI item. In some configurations, in response to identifying a symbolic gesture, the foldable device 301 may perform an operation on a UI item and / or a target associated with the gesture. For example, an operation associated with a symbolic gesture may move a window (UI item) to an open folder (target associated with the gesture). In some embodiments, once the foldable device 301 recognizes a gesture that exceeds a defined confidence threshold, the foldable device 301 triggers a command associated with the symbolic gesture. Therefore, the foldable device 301 may not display command-specific real-time feedback during the gesture, because the meaning of the gesture is not known until the gesture is recognized, at which time the command is executed.

[0053] Compared to symbol gestures, during a manipulation gesture, the foldable device 301 provides real-time feedback as the gesture proceeds. For example, during a manipulation gesture, the foldable device 301 can display the underlying UI items that move across area 106B in a one-to-one manner with the pointer 302. For example, a manipulation gesture can be used to perform a drag-and-drop operation, during which the dragged UI item moves across the display area in synchronization with the manipulation gesture. Another example of a manipulation gesture is a scroll gesture—e.g., moving document content up or down in synchronization with a manipulation gesture.

[0054] Symbol gestures can have defined shapes, including flick gestures, tap gestures, timed gestures, circle gestures, angle gestures, and the like. As described above, a flick gesture is a gesture that typically moves in one direction to allow a certain curvature as an error margin. A click gesture may include pressing, touching, hovering, or other activation of a UI item that lasts less than a defined time period. A timed gesture may include pressing, touching, hovering, or other activation gestures that are maintained for at least a defined time period before being released. In some configurations, multiple timed gestures may be defined for a UI item based on how long the activation gesture is maintained. A circular gesture may define a radius, direction, and degree of completion (e.g., 270 degrees). An angle gesture may define the distance before the angle, the size of the angle, and the direction of the angle.

[0055] When a gesture is first detected, the foldable device 301 can distinguish between a symbolic gesture and a manipulative gesture based on the initial gesture speed and / or acceleration. For example, if at the start of the gesture, the pointer 302 moves faster than a defined speed or accelerates faster than a defined rate within a defined amount of time, the foldable device 301 can identify the gesture as a symbolic gesture.

[0056] Once the foldable device 301 determines that the gesture is a symbolic gesture, the foldable device 301 can create a list of potential symbolic gestures that the user may be performing. For example, if the gesture begins with a file icon with a specific extension, the foldable device 301 can create a list of potential symbolic gestures supported by the file icon with the specific extension, such as moving a file, opening a file with a first application, opening a file with a second application, opening a file in a different area, etc.

[0057] The list of potential symbolic gestures can be further limited by the direction of the gesture, which region 106 the gesture begins in, the orientation and / or posture of the foldable device 301, which application or application type contains the UI item, the location of the initial touch relative to the edge of the region, the location of the initial touch relative to the hinge 108, the presence (or absence) of other UI items on or near the gesture path or in the direction of the gesture, or a combination thereof.

[0058] The foldable device 301 can then determine which possible symbolic gestures, if any, to perform. The foldable device 301 can use thresholds to determine which symbolic gesture to perform - when the pointer 302 crosses these lines, it is determined that the corresponding symbolic gesture has been performed. The foldable device 301 can create a threshold for each gesture in the set of possible gestures.

[0059] Thresholds can have different shapes, such as straight, curved, or square, and can be placed in different directions and at different distances from the start of the gesture. The specific shape, direction, and distance can be determined in part based on user configuration or default values ​​associated with each gesture. The shape, direction, and distance of the threshold can also be dynamically configured based on the presence of other thresholds - for example, to disambiguate or place one threshold above another.

[0060] The shape, direction, and distance of the threshold value may also be dynamically configured based on the UI items along the gesture or in the direction defined by the gesture. For example, if the hinge 108 is 150 pixels away from the file icon, the foldable device 301 may set a threshold in the direction of the hinge 108 to trigger a flick gesture 80 pixels away from the file icon.

[0061] Once the pointer 302 is lifted, or the gesture is determined to be complete, for example due to lack of motion, the foldable device 301 can determine which thresholds (if any) the completed gesture crosses. If the completed gesture does not cross any thresholds, the foldable device 301 can determine that a symbolic gesture that is defined by crossing a threshold is not performed. If one threshold is crossed, the corresponding symbolic gesture is identified. If more than one threshold is crossed, the foldable device 301 can select the symbolic gesture corresponding to the last threshold to be crossed. In other embodiments, each symbolic gesture can have a defined priority, and the foldable device 301 can select the symbolic gesture with the highest priority. In other embodiments, the foldable device 301 can select the symbolic gesture associated with the threshold farthest from the starting position of the gesture.

[0062] Figure 4A Pointer 302 is shown pressing location 402 of region 106B to initiate a symbolic "flick" gesture 414. As referred to herein, a pressed location may refer to a touch screen device being physically touched, e.g., with a finger, stylus, or other object. A pressed location may also refer to movement or hovering of an object over a location near the screen, e.g., hovering less than half a centimeter over location 402 for at least a defined period of time. Figure 4A A flick gesture 414 is depicted as beginning over window 406, such that window 406 becomes a parameter of a command associated with flick gesture 414. As described above, beginning a flick gesture over a window is one example of a gesture type and underlying UI item—other types of example symbolic gestures may begin over other types of UI items. Additionally or alternatively, the symbolic gesture may begin over an empty portion of region 106B, such as a desktop background or other portion of region 106B where there are no UI items in or near the region. In these cases, the corresponding operation may be a global operating system command.

[0063] Figure 4BPointer 302 is shown initiating a flick 414 by moving from location 402 of area 106B to location 408. As discussed above, foldable device 301 can distinguish flick 414 from a manipulation gesture (e.g., a drag-and-drop gesture) by the speed or acceleration of the initial movement of the gesture. Once flick 414 is identified as a symbolic gesture, foldable device 301 can establish threshold 410 between window 406 and hinge 108. The distance from location 408 to threshold 410 can be based in part on the distance from location 408 to hinge 108. The distance from location 408 to threshold 410 can also be based on a preconfigured distance that is long enough to minimize false positive gestures while being short enough to allow symbolic gestures to be easily performed. Foldable device 301 can select the shape and orientation of threshold 410, for example, a concave curve from the perspective of location 408.

[0064] Threshold 410 may be dynamically updated based on the current position, velocity, and acceleration of pointer 302. If the velocity or acceleration of pointer 302 is particularly fast, e.g., compared to a baseline for the user, threshold 410 may be moved closer to position 408 because the intent to perform a symbolic gesture is clearer. However, if the velocity or acceleration of pointer 302 is critical and / or slowing, threshold 410 may be moved away from position 408 to avoid false positives of symbolic gestures when the user actually intended to perform a manipulation gesture.

[0065] Figure 4C Pointer 302 is shown crossing threshold 410 to reach position 412. In some configurations, a symbolic gesture that crosses threshold 410 triggers a corresponding operation even before pointer 302 is lifted or the gesture otherwise ends. For example, when there is no ambiguity as to which symbolic gesture is intended—i.e., if there are no other thresholds that gesture 414 could cross—symbolic gesture 414 can be considered complete because threshold 410 is crossed.

[0066] In contrast to manipulative gestures, Figure 4CAn embodiment is shown in which the symbol gesture does not cause the window 406 to move with the pointer 302. Instead, the window 406 remains in place unless and until the flick gesture 414 is completed (and the corresponding command moves the window 406). In other embodiments, the foldable device 301 can provide visual feedback during the symbol gesture that is independent of which command the symbol gesture ultimately triggers. For example, the foldable device 301 can move the window 406 in a one-to-one manner with the pointer 302 throughout the symbol gesture. The foldable device 301 can alternatively move the window 406 in the direction of the pointer 302, but only move a limited distance or a limited percentage for the movement of the pointer 302. The foldable device 301 can also make a copy of the window 406 or an outline of the window 406, and provide visual feedback by moving the copy / outline based on the movement of the pointer 302. Additionally or alternatively, the foldable device 301 can change the size, highlight, animate, or otherwise emphasize the window 406 during the symbol gesture to indicate which UI item will be manipulated. For example, when pointer 302 performs a symbolic gesture over window 406, foldable device 301 may show that window 406 is moving, but does not provide feedback to disambiguate between related commands that the user may be performing, such as snapping the window to the edge of the area, spanning the window to cover unused portions of the area, copying, moving, etc.

[0067] In some configurations, regardless of whether the foldable device 301 provides feedback during the symbolic gesture, once the threshold 410 is crossed and the command associated with the gesture is determined, but before the gesture is completed, the foldable device 301 may provide feedback. For example, the pointer 302 may cross the threshold 410 while on its way to completing the symbolic gesture 414 at the location 412. In response to crossing the threshold 410, the foldable device 301 may determine that the command 416 for moving the window 406 is intentional. The foldable device 301 may provide an indication that the threshold 410 is crossed, for example, using a tactile, audio, or visual indication that the threshold 410 is crossed. For example, if the threshold 410 is visible, the foldable device 301 may flash, highlight, or otherwise emphasize the threshold 410 when the pointer 302 crosses the threshold 410. Regardless of whether the threshold 410 is visible, the foldable device 301 may provide visual, audio, or tactile feedback when the threshold 410 is crossed, for example, by bouncing, shaking, changing size, or otherwise changing the appearance of the window 406. Additionally or alternatively, the foldable device 301 can indicate which command will be executed on the window 406 with visual, audio, or tactile feedback. 4A to 4DIn the example shown, once pointer 302 crosses threshold 410, foldable device 301 can display an outline of window 406 in the target area. Once gesture 414 ends at location 412, foldable device 301 can perform a move operation 416 by moving window 406 to the location where the outline is displayed.

[0068] Figure 4D Pointer 302 is shown released from region 106B, completing flick gesture 414. In this example, the operation corresponding to flick gesture 414 is operation 416, moving target window 406 to region 106A. In this manner, window 406 is moved to region 106A without flick gesture 414 entering region 106A.

[0069] As described above, if threshold 410 is crossed within a defined amount of time, and / or if threshold 410 is crossed without traveling beyond a defined distance, then flick gesture 414 is considered completed. Similarly, in order to be considered completed, foldable device 301 may require flick gesture 414 to be completed within a defined amount of time after crossing threshold 410 and to not deviate from the direction of flick gesture 414 by more than a defined number of degrees after crossing threshold 410. By requiring flick gesture 414 to be released quickly and without a significant change in direction, foldable device 301 has more confidence that flick gesture 414 is the intended gesture, i.e., flick gesture 414 is not actually the beginning of a manipulation gesture or a different symbol gesture having a further threshold.

[0070] FIG. 5A to FIG. 5F 5 is a device user interface diagram 500 that illustrates aspects of a manipulation gesture 518 that originates above a file icon 504 in display area 106B, enters a dynamically determined target range 510A, and causes an operation to be performed on a corresponding file in area 106A. As described above, a manipulation gesture is a gesture that moves an item in a one-to-one manner using pointer 302. In other words, when pointer 302 moves around area 106B to perform manipulation gesture 518, UI items such as file icon 504 move with it.

[0071] Figure 5A Pointer 302 is shown initially touching area 106B at location 502. Location 502 is over file icon 504, and thus file icon 504 can be one of the parameters passed to the operation corresponding to manipulation gesture 518.

[0072] Figure 5BThe pointer 302 is shown with a file icon 504 being dragged to a location 508 along a path 506. In some embodiments, the pointer 302 is initially dragged at a sufficiently slow speed or acceleration to be considered a manipulation gesture. Once the foldable device 301 determines that the pointer 302 is moving at a speed and / or acceleration consistent with a manipulation gesture, the foldable device 301 begins moving the file icon 504 along the path 506 in a one-to-one manner with the pointer 302.

[0073] As pointer 302 drags file icon 504 along path 506, foldable device 301 can iteratively scan regions 106B and 106A (including region 106A itself) to find destinations and / or potential operations that are consistent with path 506 and the current location of file icon 504. 4A to 4D With respect to the identification of symbolic gestures discussed above, the foldable device 301 can narrow down the selection of destinations and / or operations based on the current (i.e., tangential) direction of the path 506, potential destinations on or near the path 506, potential destinations in the direction of the path 506, the edges of the region 106B and the hinge 108, and the like.

[0074] As shown, path 506 has turned upward toward hinge 108, and therefore region 106A has been identified as a potential destination. Therefore, foldable device 301 has created range 510A as a position where file icon 504 can be dragged to move file icon 504 to region 106A. The position, shape, and orientation of range 510 can be determined based on multiple factors, including the speed and acceleration of the gesture, the direction in which the gesture is performed, especially when the direction is toward a potential target such as hinge 108 or a folder, and the linear measurement of the gesture, that is, whether there is a clear dominant direction, rather than moving in a wave or zigzag manner. Range 510 can take any shape, such as an ellipse, rectangle, or irregular shape. When other potential destinations can obscure the target of the operation, range 510 may present an irregular shape. For example, a range 510 that was originally rectangular can have a circular shape cut out from one corner to prevent the overlap between range 510 and a folder.

[0075] Figure 5C Pointer 302 is shown to have moved further along path 506, continuing to turn upward toward region 106A. Thus, foldable device 301 has identified range 510B, so that if gesture 518 ends in range 510B, foldable device 301 moves file icon 504 to region 106A.

[0076] FIG. 5D to FIG. 5F Three possible results of dropping file icon 504 into range 510B are shown. Figure 5DThe file icon 504 is shown being moved to area 106A, e.g., to the desktop of area 106A. In some configurations, the manipulation gesture 518 may instead create a copy of the underlying file in area 106A, and / or create a shortcut in area 106A to the underlying file associated with the file icon 504. Which action is performed may be determined by user configuration, default values, command modifiers, etc.

[0077] The foldable device 301 may also be configured to perform more complex operations. For example, Figure 5E The file icon 504 is shown being moved to an open folder 514 in the region 106B. In some configurations, the presence of the open folder 514 in the region 106B causes the position and / or orientation of the range 510B to be relative to the Figure 5D For example, since folder 514 is located to the right of region 106B, range 510B may be moved further to the right. Figure 5E Also depicted is a second folder 515 in area 106B. A second scope 510C is active in response to folder 515.

[0078] Fig. 5F Another operation that foldable device 301 can perform on file icon 504 when manipulation gesture 518 ends in area 510B is shown: opening an application associated with file icon 504 in area 106A.

[0079] Figure 6 6 is a device user interface diagram 600 showing aspects of a target range that is dynamically determined when a file icon 604 is dragged over a potential destination 612 in area 106B. In some embodiments, Figure 6 Pointer 302 is shown starting manipulation gesture 606 of file icon 604 at location 602, dragging file icon 604 to location 608. If folder 612 does not exist or is not on the path of manipulation gesture 606, foldable device 301 may have sufficient confidence in the intended destination of file icon 604 to create scope 610.

[0080] However, the presence of folder 612 as a possible destination for file icon 604 creates ambiguity as to the intended destination of file icon 604. Therefore, foldable device 301 will create range 614 outside of folder 602 to remove any ambiguity. In another embodiment, range 614 may be created with a cutout ("carve-out") at a location closer to location 602 to avoid folder 612. If manipulation gesture 606 continues to range 614 in which file icon 604 is dropped, foldable device 301 may perform the above-described FIG. 5D to FIG. 5F One or more of the operations depicted in .

[0081] Figure 7 700 is a device user interface diagram illustrating aspects of a target range that is dynamically determined when a potential target 702 is exited along an expected gesture path. In some configurations, an open folder 702 is ambiguous with respect to a predicted destination of a file icon 604. Although the open folder 702 is not on the path 606 of the file icon 604, nor adjacent to or near the file icon 604, the open folder 702 is on the expected gesture path 704 of the file icon 604.

[0082] Expected gesture path 704 can be determined based on the direction of file icon 604, ie, the tangent direction vector when generating expected gesture path 704. Expected gesture path 704 can also be determined based on the overall direction of manipulation gesture 606.

[0083] Figure 8 FIG. 800 is a device user interface diagram showing aspects of a prediction 802 of an operation that a user intends to perform based in part on an extrapolation of a gesture 606. In some configurations, the visual effect 802 is a prediction based on an extrapolation of the gesture 606, similar to the above in conjunction with Figure 7 804. In some configurations, the foldable device 301 creates a visual effect 802 to preview what action will be performed on the file icon 604 if the gesture 606 ends at the current location. As shown, the foldable device 301 is displaying the end manipulation gesture 606 because it will invoke the action of moving the file icon 604 to open the folder 702.

[0084] The shape of the visual effect is not necessarily like the underlying UI item. For example, the visual effect indicating the predicted destination for the file icon 604 can have a rectangular shape instead of the outline of the file icon 604.

[0085] In addition, the visual effect can change shape, position, and size to reflect more or less confidence in the prediction. For example, the size of the visual effect 802 can increase as the gesture 606 approaches the open folder 702 to indicate that the confidence in the prediction increases.

[0086] Visual effects can also be used to communicate that the foldable device has interpreted the user's intent, but that a particular destination is invalid. For example, if the open folder 702 is read-only, the visual effect 802 may appear in the open folder 702, but "bounce," "shake," or otherwise indicate through animation or other appearance that the moving file icon 604 for the open folder 702 is invalid.

[0087] In some configurations, if the pointer 302 hovers over a specific manipulation range while performing a gesture, the foldable device 301 generates a visual pulse.

[0088] Fig. 9 900 is a device user interface diagram illustrating aspects of a prediction of an operation that a user may perform based in part on an analysis of a gesture. In some embodiments, manipulation gesture 606 has continued past open folder 702, thereby eliminating any potential ambiguity regarding the destination of file icon 604 caused by open folder 702. Instead, pointer 302 has continued manipulation gesture 606 to extent 714, which is associated with an operation for opening a file associated with file icon 604 in area 106A.

[0089] In some configurations, prediction 902 is a visual effect that indicates what will happen to file icon 604 if manipulation gesture 606 ends in range 714. In some configurations, prediction 902 outlines what an application for opening a file associated with file icon 604 will look like. For example, prediction 902 may include the location and size of the application. In other embodiments, foldable device 301 may open the file associated with file icon 604 and display the actual file contents in prediction 902.

[0090] Fig.10 1000 is a device user interface diagram illustrating aspects of a description of the amount of confidence in a prediction 902 of an operation that a user may perform. In some configurations, the confidence in the prediction 902 is indicated by highlighting, bolding, increasing, or otherwise distorting the appearance of the application depicted in the prediction 902. Darker, thicker colors may indicate higher confidence, although lighter, brighter colors may similarly be used to indicate more confidence in the prediction. Additionally or alternatively, other visual indications such as line thickness, font size, opacity level, and similar visual cues may alert the user to more or less confidence in the outcome of the prediction of the ending manipulation gesture 606.

[0091] In some configurations, the foldable device 301 can indicate how the amount of confidence in the prediction 902 changes throughout the manipulation gesture. For example, the prediction 902 can gradually fade in, e.g., the outline becomes darker or the drawing of the prediction becomes opaque as the foldable device 301 becomes more confident in the user's intention. For example, once the file icon 604 has been dragged from position 602 to position 1002, the prediction 902 can be displayed as a light outline to indicate the slight confidence that the user intends to drag the icon 604 to the range 714. As the file icon 604 is dragged further toward position 1004, the outline can become darker because the confidence in the range 714 increases with the destination. If the file icon 604 is dragged away from the range 714, the confidence in the range 714 decreases with the destination, and the prediction 902 can fade out accordingly.

[0092] Position 1002 is an example of a position where foldable device 301 can determine with some confidence that file icon 604 will be dragged to range 714 and therefore display prediction 902 in a light outline. This determination can be based on the velocity and acceleration of pointer 302, the distance of location 1002 from other potential destinations (such as folder 702), the distance pointer 302 travels from location 602 to location 1002, changes in the direction of the gesture over time, whether the gesture moves toward a different display area, etc.

[0093] In different situations, foldable device 301 may have different confidence levels at location 1002, i.e., that the gesture will end in range 714. For example, if the average speed of the gesture from location 602 to location 1002 is low, there may be less confidence in range 714 as the destination because the user may be hesitant, or the gesture may still turn left toward folder 702. In this case, prediction 902 may not be displayed at all. Conversely, if the average speed of the gesture is higher, foldable device 301 has more confidence that the gesture will bypass folder 702, and prediction 902 may be displayed with a darker outline or a more opaque prediction rendering.

[0094] The distance the gesture has traveled so far is another factor. If the gesture has not traveled as much distance—that is, if location 602 is closer to location 1002—then foldable device 301 may have less confidence in the overall direction of the gesture. If there is less confidence in the direction of the gesture, foldable device 301 may have less confidence that range 714 is the destination.

[0095] When foldable device 301 has less confidence that the gesture will end in folder 702, foldable device 301 can have more confidence that the gesture will end in location 714. Foldable device 301 can calculate that the gesture will not end in folder 702 based on the factors listed above for predicting that the gesture will end in range 714. For example, the distance from location 602 to location 1002, the consistency of the direction of the gesture, whether the direction of the gesture moves toward folder 702 and how much, whether the gesture meaningfully slows down by more than a defined amount when approaching folder 702, etc.

[0096] In some configurations, foldable device 301 may display multiple predictions of where the gesture will end. For example, foldable device 301 may detect two possible destinations for file icon 604 with sufficient confidence to display predictions—folder 702 and range 714. In this case, foldable device 301 may display prediction 802 within folder 702 and prediction 902 within area 106A. As the gesture continues, the confidence in one or more of the predictions may increase, decrease, or remain the same, causing the prediction to be emphasized, de-emphasized, or remain the same.

[0097] In some configurations, the confidence in prediction 902 is interpreted based on how the manipulation gesture directly reaches range 714. Directly reaching range 714 can generate more confidence in the prediction than if the manipulation gesture 606 went around ("wend around") in multiple directions before encountering range 714. Another basis for the confidence score for prediction 902 is how far the pointer 302 has traveled in range 714 - just skimming the surface of range 714 may provide less confidence than entering the center of range 714. Another basis for the confidence score is whether the speed and acceleration of the pointer 302 indicates that the pointer 302 is likely to slow down and remain in range 714 (increasing confidence) or whether the speed and acceleration of the pointer 302 is likely to accelerate and / or pass through range 714 (reducing confidence).

[0098] FIG. 11A to FIG. 11D is a device user interface diagram 1100 showing aspects of a symbolic "flick" gesture. In some configurations, the foldable device 301 is oriented in landscape mode to turn each individual region into portrait mode.

[0099] Fig.11A Pointer 302 is shown having pressed location 1104 of region 106A and slid to location 1108 of region 106A. Pointer 302 has moved away from location 1104 with sufficient speed and acceleration for foldable device 301 to identify the movement as a symbolic gesture. Different speeds and lengths of symbolic gesture 1108 may be associated with different operations. Fig.11A Also shown is another example of a prediction 1112 of an outcome of an operation associated with the symbolic gesture 1108 .

[0100] Fig. 11B The symbolic gesture 1108 is shown crossing a threshold 1110, after which the pointer 302 is lifted from the region 106A at a position 1114. The foldable device 301 can then calculate the average speed of the symbolic gesture and the distance traveled by the symbolic gesture. The speed and distance of the symbolic gesture can be used to distinguish between different gestures, each of which can be associated with a different operation. For example, Fig. 11B It is shown that when the symbol gesture 1108 ends at the position 1114, the operations performed move the window 1102 to the area 106B.

[0101] Fig. 11C The symbol gesture 1108 is shown crossing the threshold 1110, after which the pointer 302 is lifted from the region 106A at a position 1116. In some configurations, the position 1116 is further away from the position 1104 than the position 1114. The foldable device 301 can then calculate the average speed of the symbol gesture and the distance traveled by the symbol gesture. Figure 1B As discussed, the speed and distance of a symbolic gesture can be used to distinguish different operations associated with the gesture. For example, Fig. 11C It is shown that when symbol gesture 1108 ends at position 1116, the operations performed move window 1102 to area 106B and maximize window 1102 to the entire area 106B. Associating the operations for moving and maximizing window 1102 with a longer gesture may be intuitive to the user because moving and maximizing is more than just moving.

[0102] Fig.11D Symbolic gesture 1108 is shown crossing threshold 1110 and proceeding to position 1118, which is further away from position 1104. The operation corresponding to the longer symbolic gesture is to move window 1102 to region 106B and have window 1102 span both regions 106A and 106B.

[0103] Fig.12 1 is a flow diagram illustrating a routine 1200 illustrating aspects of the operation of a foldable computing device 102 for predictive gesture optimization for moving an object across a display boundary. The routine 1200 begins at operation 1202, where the foldable device 301 determines an initial position 502 of a touch screen gesture 518 within the first area 106A of the foldable device 301 in the manner described above. The routine 1200 then proceeds to operation 1204, where the foldable computing device 102 determines a UI item 504 at the initial position 502 in the manner described above.

[0104] The routine 1200 then proceeds to operation 1206, where the foldable device 301 creates and dynamically updates the range 510 in the display area 106A while the gesture 518 is in progress, in the manner described above. The routine 1200 then proceeds to operation 1208, where the foldable device 301 determines whether the gesture 518 is complete, in the manner described above. If the foldable device 301 determines that the gesture 518 is not complete, the routine 1200 proceeds to operation 1206. If the foldable device 301 determines that the gesture 518 is complete, the routine 1200 proceeds to operation 1210, where the foldable device 301 determines the location 512 in which the gesture 518 is complete, in the manner described above.

[0105] From operation 1210, the routine 1200 proceeds to operation 1212, where the foldable device 301 determines whether the gesture 518 is completed in the range 510 in the manner described above. If the foldable device 301 determines that the gesture 518 is not completed in the range 510, the routine 1200 proceeds to operation 1216, where the foldable device 301 performs an operation on the UI item 504 in the first display area 106A in the manner described above. However, if the foldable device 301 determines that the gesture 518 is completed in the range 510 of the area 106A, the routine 1200 proceeds to operation 1214, where the foldable device 301 performs an operation associated with the gesture 518 on the UI item 504 in the display area 106B in the manner described above.

[0106] The routine 1200 then proceeds to operation 1218, where operation 1218 ends.

[0107] Composite symbol and gesture language for multi-screen windowing

[0108] Fig.13A and Fig. 13B 1300 is a device user interface diagram showing aspects of a combination of a manipulation gesture 1306 and a symbol gesture 1310 for opening a document 1304. Manipulation gestures and symbol gestures can be combined to perform more advanced, more complex gestures. For example, if a user is dragging a window and quickly flicks downward, the foldable device 301 can minimize the window to a taskbar associated with the area where the flick began - regardless of where the window was originally located. Similarly, a drag-and-drop manipulation gesture of a window can be interrupted by a flick across to move the window to an adjacent area.

[0109] In another embodiment, a drag-and-drop manipulation gesture can be interrupted with an upward flick to maximize the window in the area where the flick occurred, rather than necessarily where the window was prior to the manipulation gesture. In this way, the context created by the manipulation gesture is used as a modifier when interpreting the symbolic gesture. Other examples of symbolic gestures include flicking upward to open a new instance of an application. Opening a new instance of an application can also apply to icons on the desktop or icons in the taskbar that represent an already running instance of an application.

[0110] Fig.13A Pointer 302 is shown having touched location 1302 on region 106A of foldable device 301 to begin compound gesture 1305. Document icon 1304 is initially located at location 1302 but is moved to location 1308 by manipulation gesture component 1306 of compound gesture 1305, which ends in location 1308 and symbol gesture component 1310 of compound gesture 1305 begins in location 1308. Pointer 302 is depicted as sliding upward past threshold 1312 to location 1314 while performing symbol gesture component 1310.

[0111] Fig. 13B Pointer 302 is shown released from area 106A to complete symbolic gesture component 1310 and composite gesture 1305 including manipulation gesture component 1306 and symbolic gesture component 1310. The operation associated with symbolic gesture component 1310 is to open the document associated with document icon 1304 in area 106A (i.e., the area in which symbolic gesture 1310 was initiated). Although FIG. 13 depicts symbolic gesture component 1310 crossing threshold 1312, it is similarly contemplated that manipulation gesture component 1306 may cross threshold 1312 while symbolic gesture component 1310 is used to determine or modify the operation to be performed in display area 106A.

[0112] Fig.14A and Fig. 14B is a device user interface diagram 1400 showing aspects of a combination of a manipulation gesture component 1406 and a symbol gesture component 1410 of a compound gesture 1405 for opening a document 1404 on a different display area 106B.

[0113] Fig.14APointer 302 is shown having touched location 1302 on region 106A of foldable device 301 to begin manipulation of gesture component 1406 of composite gesture 1405. Document icon 1404 is initially located at location 1302 but is moved by manipulation gesture component 1406 to location 1408 in region 106B. At location 1408, composite gesture 1405 transitions from manipulation gesture component 1406 to symbol gesture component 1410. In some configurations, the end of manipulation gesture component 1406 and the beginning of symbol gesture component 1410 are identified based on a change in direction, velocity, and / or acceleration exceeding a defined threshold. Pointer 302 is depicted as sliding upward beyond threshold 1412 to location 1414 while performing symbol gesture component 1410.

[0114] Fig. 14B Pointer 302 is shown released from region 106B to complete symbol gesture component 1410 and composite gesture 1405 including manipulation gesture component 1406 and symbol gesture component 1410. The operation associated with symbol gesture component 1410 is to open the document associated with document icon 1404 in region 106B (i.e., the region in which symbol gesture component 1410 was initiated).

[0115] Fig.15 1 is a flow chart 1500 showing a routine that illustrates aspects of a composite symbol and manipulation gesture language for multi-screen windowing. The routine 1500 begins at operation 1502, where the foldable device 301 determines an initial position 1302 of a touch screen manipulation gesture 1406 within the first area 106A of the foldable device 301 in the manner described above. The routine 1500 then proceeds to operation 1504, where the foldable computing device 102 determines a UI item 1404 at the initial position 1302 in the manner described above.

[0116] From operation 1504, the routine 1500 proceeds to operation 1506, where the foldable device 301 identifies a transition from the manipulation gesture component 1406 to the symbolic gesture component 1410 in the manner described above. The routine 1500 then proceeds to operation 1508, where the foldable device 301 determines a location on the region 106A or region 106B where the manipulation gesture 1406 transitions to the symbolic gesture component 1410 in the manner described above.

[0117] From operation 1508, routine 1500 proceeds to operation 1510, where foldable device 301 performs an operation associated with symbolic gesture component 1410 on UI item 1404. The selected operation may be modified by manipulation gesture component 1406, e.g., the location of the transition from manipulation gesture component 1406 to symbolic gesture component 1410 may affect the command applied to UI item 1404 by symbolic gesture component 1410. For example, if manipulation gesture component 1406 begins in region 106A and transitions to symbolic gesture component 1410 in region 106B, the effect of symbolic gesture component 1410 may be different than the transition to symbolic gesture component 1410 that occurs in region 106A. For example, the region 106 in which the transition occurs may determine in which region a window is maximized. Routine 1500 then proceeds to operation 1512, where it ends.

[0118] Fig.16 is a computer architecture diagram showing an illustrative computer hardware and software architecture of a computing device that can implement the various techniques presented herein. In particular, Fig.16 The illustrated architecture may be used to implement the articulated device 102 and the bendable device 202 described herein. The illustrated architecture may also be used to implement other types of computing systems.

[0119] Fig.16 The illustrated computer 1600 includes a central processing unit 1602 ("CPU"), a system memory 1604 (including random access memory 1606 ("RAM") and read-only memory ("ROM") 1608), and a system bus 1610 that couples the memory 1604 to the CPU 1602. A basic input / output system ("BIOS" or "firmware") containing basic routines to help transfer information between elements within the computer 1600, such as during startup, may be stored in the ROM 1608. The computer 1600 also includes a mass storage device 1612 for storing an operating system 1622, application programs 1630, and other types of programs. The functions described above are implemented by one or more of these programs in various configurations. The mass storage device 1612 may also be configured to store other types of programs and data.

[0120] The mass storage device 1612 is connected to the CPU 1602 through a mass storage controller (not shown) connected to the bus 1610. The mass storage device 1612 and its associated computer-readable media provide non-volatile storage for the computer 1600. Although the description of computer-readable media contained herein refers to a mass storage device, such as a hard disk, a CD-ROM drive, a DVD-ROM drive, or a USB memory key, it should be understood by those skilled in the art that the computer-readable media can be any available computer storage media or communication media that can be accessed by the computer 1600.

[0121] Communication media includes computer readable instructions, data structures, program modules, or other data in a modulated data signal (such as a carrier wave or other transmission mechanism), and includes any delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics changed or set in some way so as to encode information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.

[0122] By way of example and not limitation, computer storage media may include volatile and nonvolatile removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technology, CD-ROM, digital versatile disk ("DVD"), HD-DVD, BLU-RAY, or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the computer 1600. In the claims, the phrase "computer storage medium" and its variations do not include waves or signals themselves or communication media.

[0123] According to various configurations, the computer 1600 can operate in a network environment using logical connections to remote computers through a network such as network 1620. The computer 1600 can be connected to the network 1620 through a network interface unit 1616 connected to the bus 1610. It should be understood that the network interface unit 1616 can also be used to connect to other types of networks and remote computer systems. The computer 1600 may also include an input / output controller 1618 for receiving and processing input from a variety of other devices, including a keyboard, mouse, touch input, digital pen, or physical sensors such as cameras and biometric sensors.

[0124] The computer 1600 may also be configured with a suitable video output device that may provide output to one or more display screens, such as those described above. One or more of the displays may be touch-sensitive displays configured to detect the presence and location of a touch. Such displays may be resistive touch screens, capacitive touch screens, surface acoustic wave touch screens, infrared touch screens, optical imaging touch screens, dispersive signal touch screens, acoustic pulse recognition touch screens, or may utilize any other touch screen technology. In some configurations, a touch screen is incorporated as a transparent layer on top of the display to enable a user to interact with an object or other information presented on the display using one or more touches.

[0125] The touch-sensitive display can be configured to detect discrete touches, single-touch gestures, and / or multi-touch gestures. For convenience, these are collectively referred to herein as "gestures." A number of gestures will now be described. It should be understood that these gestures are illustrative and are not intended to limit the scope of the appended claims.

[0126] In some configurations, the computer 1600 supports a tap gesture in which a user taps the display once. A double-click gesture in which a user taps the display twice may also be supported. The double-click gesture may be used for a variety of reasons, including but not limited to zooming in or out in stages. In some configurations, the computer 1600 supports a click-and-hold gesture in which a user clicks and maintains contact for at least a predefined time. The click-and-hold gesture may be used for a variety of reasons, including but not limited to opening a context-specific menu.

[0127] In some configurations, the computer 1600 supports a pan gesture in which a user places a finger on the display and maintains contact with the display while moving their finger. The pan gesture may be used for a variety of reasons, including but not limited to moving through a screen, image, or menu at a controlled rate. Multiple finger pan gestures are also contemplated.

[0128] In some configurations, the computer 1600 supports a flick gesture in which a user slides a finger in the direction the user wants the screen to move. The flick gesture may be used for a variety of reasons, including, but not limited to, scrolling a menu or page horizontally or vertically. In some configurations, the computer 1600 supports a pinch and stretch gesture in which a user makes a pinching motion with two fingers (e.g., a thumb and index finger) or moves the two fingers apart. The pinch and stretch gesture may be used for a variety of reasons, including, but not limited to, gradually zooming in or out on a website, map, or picture.

[0129] Although the above gestures have been presented with reference to using one or more fingers to perform the gestures, other accessories such as a digital pen may be used to interact with the computing device 1600. Therefore, the above gestures should be understood to be illustrative and should not be construed as limiting in any way.

[0130] It should be understood that the software components described herein, when loaded into the CPU 1602 and executed, can transform the CPU 1602 and the entire computer 1600 from a general-purpose computing device to a customized special-purpose computing device for facilitating the functions described herein. The CPU 1602 can be composed of any number of transistors or other discrete circuit elements, which can present any number of states individually or collectively. More specifically, the CPU 1602 can operate as a finite state machine in response to executable instructions contained in the software modules disclosed herein. These computer executable instructions can transform the CPU 1602 by specifying how the CPU 1602 transitions between states, thereby transforming the transistors or other discrete hardware elements that constitute the CPU 1602.

[0131] Encoding the software modules presented herein may also transform the physical structure of the computer-readable medium presented herein. In different implementations of this description, the specific transformation of the physical structure depends on various factors. Examples of such factors include, but are not limited to, the technology used to implement the computer-readable medium, whether the computer-readable medium is characterized as primary or secondary storage, etc. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed herein may be encoded on the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software may transform the state of transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software may also transform the physical state of such components in order to store data thereon.

[0132] As another example, the computer readable medium disclosed herein can be implemented using magnetic or optical technology. In such an implementation, when the software is encoded therein, the software presented herein can transform the physical state of the magnetic or optical medium. These transformations can include changing the magnetic properties of a specific location in a given magnetic medium. These transformations can also include changing the physical characteristics or properties of a specific location in a given optical medium to change the optical properties of those locations. Without departing from the scope and spirit of this specification, other transformations of the physical medium are possible, and the aforementioned examples are provided only for ease of discussion.

[0133] In view of the above, it should be understood that many types of physical transformations occur in the computer 1600 in order to store and execute the software components presented herein. It should also be understood that Fig.16The architecture shown for computer 1600 or similar architectures may be used to implement other types of computing devices, including handheld computers, video game devices, embedded computer systems, mobile devices (such as smart phones, tablet computers, and AR / VR devices), and other types of computing devices known to those skilled in the art. It is also contemplated that computer 1600 may not include Fig.16 All components shown can be included in Fig.16 Other components not explicitly shown in the figure or the like may be used Fig.16 The architecture shown is a completely different architecture.

[0134] It should be understood that Fig.16 The computing architecture shown has been simplified for ease of discussion. It should also be understood that the computing architecture shown may include and utilize more computing components, devices, software programs, network devices, and other components not specifically described herein.

[0135] The disclosure set forth herein also covers the subject matter set forth in the following clauses:

[0136] Example 1: A computer-implemented method performed by a foldable computing device, comprising: identifying the start of a user interface gesture in a first display area of ​​the foldable computing device, wherein the user interface gesture is associated with a user interface item, and wherein the foldable computing device includes a second display area; detecting the end of the user interface gesture in the first display area of ​​the foldable computing device; and in response to the start and end of the user interface gesture in the first display area, performing an operation associated with the user interface item and the second display area.

[0137] Example 2: The computer-implemented method according to Example 1 further includes: limiting a gesture target range within the first display area; performing an operation if the user interface gesture ends within the gesture target range; and performing a different operation associated with the first display area or performing no operation if the user interface gesture ends in the first display area outside the gesture target range.

[0138] Example 3: According to the computer-implemented method of Example 2, it also includes: during the gesture, when the gesture is within the gesture target range, displaying a visual effect in the second display area, the visual effect visualizing the operation that will be performed if the gesture ends.

[0139] Example 4: According to the computer-implemented method of Example 2, it also includes: dynamically adjusting the position, size, orientation or shape of the gesture target range based on the position, direction or speed of the user interface gesture during the user interface gesture.

[0140] Example 5: The computer-implemented method of Example 4, wherein when the speed of the user interface gesture is determined to exceed a defined threshold, the gesture target range is moved closer to the start of the user interface gesture.

[0141] Example 6: The computer-implemented method of Example 2, wherein the position, size, or shape of the gesture target range is determined based in part on the position of the start of the user interface gesture and the position of the second display area.

[0142] Example 7: The computer-implemented method of Example 2, wherein the operation is performed in response to the user interface gesture entering the gesture target range and before the user interface gesture has ended.

[0143] Example 8: A foldable computing device comprising: one or more processors; and at least one non-transitory computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the foldable computing device to: identify the start of a user interface gesture in a first display area of ​​the foldable computing device, wherein the user interface gesture is associated with a user interface item, and wherein the foldable computing device includes a second display area; after the gesture is initiated, determine a threshold on the first display area based on a start location of the user interface gesture and a location of the second display area; determine that the user interface gesture crosses the threshold; detect an end of the user interface gesture within the first display area; and in response to determining that the user interface gesture crosses the threshold and ends in the first display area, perform operations associated with the user interface item and the second display area.

[0144] Example 9: The foldable computing device of Example 8, wherein the operation associated with the second display area is performed in response to the user interface gesture being performed in less than a defined amount of time.

[0145] Example 10: The foldable computing device of Example 8, wherein the foldable computing device has a gesture based on an orientation of the first display area relative to the second display area, and wherein an operation associated with the second display area is selected based in part on the gesture.

[0146] Example 11: A foldable computing device according to Example 8, wherein the size, shape, or position of the threshold is dynamically updated during the user interface gesture based on at least one of: the position, direction, or speed of the user interface gesture or the historical position, direction, or speed of the user interface gesture.

[0147] Example 12: The foldable computing device of Example 8, wherein the threshold is made visible on the first display area when the user interface gesture has traversed at least a defined percentage of the distance between the start of the user interface gesture and the threshold toward the threshold.

[0148] Example 13: According to Example 8, the foldable computing device further includes: determining whether the second display area includes a user interface gesture target; and when the second display area includes the user interface gesture target, setting the position of the threshold to be closer to the start of the user interface gesture.

[0149] Example 14: According to Example 8, the foldable computing device also includes: identifying multiple gesture targets in the second display area; defining a threshold in the first display area for each gesture target in the multiple gesture targets; and selecting an operation based on which threshold of the multiple thresholds the user interface gesture crosses.

[0150] Example 15: A foldable computing device according to Example 14, wherein a user interface gesture is directed to two or more gesture targets among a plurality of gesture targets, wherein one of the two or more gesture targets among the plurality of gesture targets is closer to the user interface gesture, and one of the two or more gesture targets among the plurality of gesture targets is farther away from the user interface gesture, wherein when a speed of the gesture is below a defined threshold, the closer gesture target is selected, and wherein when the speed of the gesture exceeds the defined threshold, the farther gesture target is selected.

[0151] Example 16: A computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a foldable computing device, cause the foldable computing device to: identify the start of a user interface gesture in a first display area of ​​the foldable computing device, wherein the user interface gesture is associated with a user interface item, and wherein the foldable computing device includes a second display area; determine a gesture target range on the first display area based on a start location of the user interface gesture and a location of the second display area; determine that the user interface gesture ends within the gesture target range; and in response to determining that the user interface gesture ends within the gesture target range, perform an operation associated with the user interface item and the second display area.

[0152] Example 17: A non-transitory computer-readable storage medium according to Example 16, wherein an operation is performed only if the user interface gesture ends within a defined time period after entering a gesture target range and only if the user interface gesture ends after entering the gesture target range without changing the direction by more than a threshold angle.

[0153] Example 18: The non-transitory computer-readable storage medium according to Example 16 also includes: identifying potential gesture targets on the first display area; in response to the gesture entering within a defined distance of one or more potential gesture targets on the first display area, moving the gesture target range closer to the second display area.

[0154] Example 19: A non-transitory computer-readable storage medium according to Example 16, wherein the user interface gesture includes a compound gesture, the compound gesture begins with a manipulation gesture and ends with a symbol gesture, the manipulation gesture moves the user interface item and the symbol gesture does not move the user interface item, and wherein the symbol gesture determines or modifies which of a plurality of operations associated with the second display area is performed.

[0155] Example 20: The non-transitory computer-readable storage medium of Example 19, wherein before the symbolic gesture is performed, the manipulation gesture moves the user interface item to the second display area.

[0156] Based on the foregoing, it should be understood that techniques for predictive gesture optimization for moving objects across display boundaries have been disclosed herein. Although the subject matter presented herein has been described in language specific to computer structural features, methods and conversion actions, specific computing machines, and computer-readable media, it should be understood that the subject matter set forth in the appended claims is not necessarily limited to the specific features, actions, or media described herein. Rather, specific features, actions, and media are disclosed as example forms of implementing the claimed subject matter.

[0157] The above subject matter is provided by way of illustration only and should not be construed as limiting. Various modifications and changes may be made to the subject matter described herein without following the example configurations and applications illustrated and described, and without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A computer-implemented method performed by a foldable computing device, comprising: identifying a start of a user interface gesture in a first display area of ​​the foldable computing device, wherein the user interface gesture is associated with a user interface item, and wherein the foldable computing device includes a second display area that is separated from the first display area via folding; Limiting a gesture target range within the first display area; dynamically adjusting a position, size, orientation, or shape of the gesture target range based on a position, direction, or speed of the user interface gesture during the user interface gesture, wherein the gesture target range is moved closer to the start of the user interface gesture when the speed of the user interface gesture is determined to exceed a defined threshold; detecting an end of the user interface gesture in the first display area of ​​the foldable computing device; If the user interface gesture ends within the gesture target range, performing a display operation associated with the user interface item and the second display area; and If the user interface gesture ends outside of the gesture target range, performing a different operation associated with the first display area or performing no operation.

2. The computer-implemented method of claim 1 , further comprising: During the user interface gesture, while the user interface gesture is within the gesture target range, displaying a visual effect in the second display area to preview the display operation that will be performed if the user interface gesture ends.

3. The computer-implemented method of claim 1, wherein the position, the size, or the shape of the gesture target range is determined based in part on a position of the start of the user interface gesture and a position of the second display area.

4. A foldable computing device comprising: one or more processors; as well as At least one non-transitory computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by the one or more processors, cause the foldable computing device to implement the method of any of the preceding claims. 5 . The foldable computing device of claim 4 , wherein the display operation is performed in response to the user interface gesture being performed in less than a defined amount of time.

6. The foldable computing device of claim 4, wherein the foldable computing device has a gesture based on an orientation of the first display area relative to an orientation of the second display area, and wherein the display operation associated with the second display area is selected based in part on the gesture.

7. The foldable computing device of claim 4, wherein the size, shape, or position of the threshold is dynamically updated during the user interface gesture based on at least one of: a direction of the user interface gesture or a speed of the user interface gesture.

8. The foldable computing device of claim 4 , wherein the position of the threshold is made visible on the first display area when the user interface gesture traverses at least a defined percentage of the distance between the starting position of the user interface gesture and the position of the threshold toward the position of the threshold.

9. The foldable computing device of claim 4, wherein the instructions further cause the foldable computing device to: determining whether the second display area includes a user interface gesture target; and When the second display area includes the user interface gesture target, the position of the threshold is set closer to the start position of the user interface gesture.

10. The foldable computing device of claim 4, wherein the instructions further cause the foldable computing device to: identifying a plurality of gesture targets in the second display area; defining a position of a threshold in the first display area for each gesture target of the plurality of gesture targets; as well as The display operation is selected based on the position of which threshold value among the plurality of threshold values ​​the user interface gesture crosses.

11. The foldable computing device of claim 10 , wherein the user interface gesture is directed to two or more of the plurality of gesture targets, wherein one of the two or more of the plurality of gesture targets is closer to the user interface gesture and another of the two or more of the plurality of gesture targets is farther from the user interface gesture, wherein when a speed of the user interface gesture is lower than a defined threshold, the closer gesture target is selected, and wherein when the speed of the user interface gesture exceeds the defined threshold, the farther gesture target is selected.

12. A computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a foldable computing device, cause the foldable computing device to: identifying a start of a user interface gesture in a first display area of ​​the foldable computing device, wherein the user interface gesture is associated with a user interface item, and wherein the foldable computing device includes a second display area that is separated from the first display area via folding; Limiting a gesture target range within the first display area; dynamically adjusting a position, size, orientation, or shape of the gesture target range based on a position, direction, or speed of the user interface gesture during the user interface gesture, wherein the gesture target range is moved closer to the start of the user interface gesture when the speed of the user interface gesture is determined to exceed a defined threshold; determining a gesture target range on the first display area based on a start position of the user interface gesture; determining that the user interface gesture ends within the gesture target range; as well as In response to determining that the user interface gesture ends within the gesture target range, performing a display operation associated with the user interface item and the second display area; and If the user interface gesture ends outside of the gesture target range, performing a different operation associated with the first display area or performing no operation.

13. A computer-readable storage medium according to claim 12, wherein the display operation is performed only when the user interface gesture ends within a defined time period after entering the gesture target range, and only when the user interface gesture ends after entering the gesture target range without changing the direction by more than a threshold angle.

14. The computer-readable storage medium of claim 12, wherein the computer-executable instructions further cause the foldable computing device to: identifying potential gesture targets on the first display area; In response to the user interface gesture coming within a defined distance of one or more of the potential gesture targets on the first display area, the gesture target range is moved closer to the second display area.

15. A computer-readable storage medium according to claim 12, wherein the user interface gesture comprises a compound gesture, the compound gesture beginning with a manipulation gesture and ending with a symbolic gesture, the manipulation gesture moving the user interface item, the symbolic gesture not moving the user interface item, and wherein the symbolic gesture determines or modifies which of a plurality of operations associated with the second display area are performed.

16. The computer-readable storage medium of claim 15, wherein the manipulation gesture moves the user interface item to the second display area before the symbolic gesture is performed.

Images