Method for enhancing input of digital device by gesture

Abstract

A method for enhancing digital device input with gestures is provided. In an embodiment, a method includes capturing touch input data on a screen of a user device and non-touch gesture input data outside the screen of the user device in parallel. The method also includes determining an input command based at least in part on a combination of the parallelly captured touch input data and the non-touch gesture input data. And the method further includes affecting an operation of the user device based on the determined input command.

Description

Technical Field

[0001] The present invention generally relates to interactive input on a user device interface. Background Art

[0002] Currently, user devices (e.g., smartphones, tablet computers, laptop computers, etc.) with interactive input capabilities, such as touch screens, typically have small screens. Such small screens may limit interactive input capabilities. For example, a small screen may hinder interactions such as smooth scrolling, panning, zooming, etc. Touch interactions may require a user to perform repetitive actions to perform tasks such as multiple pinching, selecting, or scrolling motions on the touch screen of the user device. Furthermore, touch input may blur the screen of the user device, making it difficult to see the affected content.

[0003] Therefore, there is a need in the art to enhance interactive input on user device interfaces. Summary of the invention

[0004] Systems and methods according to one or more embodiments of the present invention provide enhanced interactive input such that touch input data (e.g., detected from gestures performed using the position or motion of a control object such as a hand, finger, etc.) can be fused with non-touch input data (which can be detected by sensors such as ultrasonic sensors, image or video capture capabilities (e.g., cameras), depth sensors, or the like. Advantageously, the view of an interface of a user device such as a display or screen can remain substantially unobstructed.

[0005] According to one embodiment, a method includes: capturing touch input data on a screen of a user device and non-touch gesture input data outside the screen of the user device in parallel; determining an input command based at least in part on a combination of the parallely captured touch input data and the non-touch gesture input data; and affecting the operation of the user device based on the determined input command.

[0006] According to another embodiment, a method includes: detecting, at a user device, in parallel, at least one of a touch on a screen of the user device and a gesture of at least a portion of a hand that is not touching the user device and a motion of a portion substantially in a plane parallel to the screen; determining a command based at least in part on a combination of the detected touch and the detected gesture or motion; and adjusting the operation of the user device based on the determined command. In one embodiment, the portion of the hand includes another portion of the hand for touching the screen. In another embodiment, the method further includes determining that the portion of the hand includes another portion of the hand for touching the screen before determining the command. In another embodiment, detecting the gesture or the motion further includes identifying one or more ends of the hand. In another embodiment, the hand is different from the hand for touching the screen. In another embodiment, the motion includes sliding. In another embodiment, the motion includes rotating or sweeping. In another embodiment, the portion of the hand is outside a volume including the screen and a vertical extension of the screen perpendicular to the screen. In another embodiment, detecting the touch further includes detecting multiple touches on the screen. In another embodiment, the location of the touch on the screen identifies the operation or the object to be adjusted. In another embodiment, detecting the touch further comprises detecting movement of the touch along the screen, wherein said determining the command comprises determining the command based at least in part on a combination of the detected touch movement and the detected gesture or motion. In another embodiment, the detected gesture or motion comprises a detected gesture. In another embodiment, the detected gesture or motion comprises a detected motion. In another embodiment, the detected gesture or motion comprises a detected motion performed when the portion of the hand is in the detected gesture. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A-1C is a diagram illustrating a process for enhancing touch with touchless gestures according to an embodiment of the invention.

[0008] Figure 2 A-2B is a diagram illustrating a process for enhancing touch with a touchless gesture according to another embodiment of the present invention. The figures have been revised.

[0009] Figures 3A-3C is a diagram illustrating a process for enhancing touch by using a non-touch gesture according to yet another embodiment of the present invention.

[0010] Figure 4 The flowchart is for illustrating a method for enhancing touch input by using non-touch gestures according to an embodiment of the present invention.

[0011] Figure 5The present invention is a block diagram of a system for implementing an apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0012] Systems and methods according to one or more embodiments of the present invention are provided for enhancing interactive inputs such as touch inputs with off-screen non-touch inputs (e.g., gestures), for example, via simultaneous multimodal sensor fusion. Non-touch inputs may include gestures such as poses or motions performed with a control object such as a hand, finger, etc., which may be detected by a sensor such as an ultrasonic sensor, image or video capture capabilities (e.g., a camera), a depth sensor, or the like. In this regard, sensors or technologies configured to detect non-touch inputs may be included in a user device or system and / or located on various surfaces of a user device, such as on the top, bottom, left, right, and / or behind the user device, so that non-touch data such as gestures may be captured when they are performed directly in front of the user device (on-screen) and outside the direct line of sight of the screen of the user device (off-screen). In general, off-screen non-touch inputs may also be referred to as "off-screen gestures" hereinafter, where "off-screen gestures" may refer to position or motion data of a control object such as a hand, finger, pen, etc., where the control object is not touching the user device, but is close to the user device. These "off-screen" non-touch gestures may not only be removed from the screen of the user's device, but they may include a portion of the control object being offset laterally from the device relative to the device's screen or display. For example, one may imagine a volume extending away from the device's display or screen in a direction generally perpendicular to the plane of the display or screen. An "off-screen" gesture may include a gesture where at least a portion of the control object in which the gesture is performed is outside of this volume. For example, in Figure 1 In A-1C and 2A-2B, the user's finger is "off-screen".

[0013] Concurrent interactive inputs such as touch input and off-screen non-touch input (e.g., gestures) can create advanced commands. In one embodiment, a user can touch a target item displayed on the screen of a user device and simultaneously use an off-screen gesture (e.g., a finger on the same hand or another hand) to adjust an aspect of the target item.

[0014] The off-screen non-touch gesture may include any suitable predefined gesture corresponding to adjusting a particular aspect of the target item. For example, a setting may be adjusted using a specific hand motion or position, such as a downward hand motion may correspond to reducing a brightness setting of the screen.

[0015] Touch input sense data and non-touch off-screen input sense data (hereinafter referred to as "gesture input sense data") can be captured substantially in parallel so that these input sense data sets can be combined to create one input command. That is, novel composite commands can be enabled so that input targets and variable adjustments of the input targets can be combined. It should be noted that in various embodiments, there may be minimal or negligible offset between when a gesture, such as a motion, begins or ends and when a touch input is detected. In some embodiments herein, touch input sense data and gesture input sense data can be captured substantially in parallel or simultaneously, for example to allow for less leeway between the timing of touches and gestures.

[0016] In various embodiments, gesture input sensing data may be captured by detecting the position and movement of an object. For example, the detected movement of an object such as a hand or finger may be substantially parallel or in the same plane as a surface such as a display or screen of a user device, and may be performed outside the screen rather than above the screen. Thus, obstruction of the display or screen is avoided or reduced, thus allowing the screen to be visible.

[0017] There are many examples of compound commands that can combine input targets with variable adjustments. Such examples can include adjusting one or more settings by using one or two hands, including the following: Camera: Select a target item via touch and non-touch gestures that affect the target item, such as to adjust the focus, aperture, scene mode, etc. For example, when taking a photo with the camera, the user can select an out-of-focus photo via touch (e.g., using an object such as a finger, pen, etc.) while performing a non-touch gesture that can be captured on the side of the camera to adjust the focus. In this regard, because the non-touch gesture can be performed off-screen (e.g., on the side, top, bottom, or back of the camera), the displayed photo is not blocked from the user's view. This example will be described below with respect to one embodiment. Figure 1 A-1C is described in more detail.

[0018] Settings: Select the desired setting area or icon via touch and gesture to adjust the setting or the setting associated with the icon (e.g., volume, brightness, etc.). Similar to the camera example described above, the user can select the desired setting or the icon representing the setting such as brightness via touch while performing a non-touch gesture that can be captured off-screen. The non-touch gesture can correspond to adjusting the brightness. In various embodiments, any command icon associated with a variable control can be touched, and then the variability can be adjusted by gestures such as sweeping or rotating the hand. Therefore, in some embodiments, a second menu such as a slider for adjusting brightness, volume, etc. may not be required. Because the gesture is performed and captured off-screen, the display or screen is not blocked from the user's line of sight. For example, the user device can detect touch input, such as a user's thumb touching an icon representing brightness adjustment on the screen or on one side of the bezel of the user device. At the same time, the user device can detect gestures, such as the user's fingers of the same hand (while touching the brightness icon) rotating to adjust the brightness; for example, rotating to the right can increase the brightness and rotating to the left can decrease the brightness.

[0019] Game: Select an opponent via touch and gestures such as hand position or movement to set an action to be performed on the opponent. Similar to the camera example described above, a user playing a game on a user device can select an opponent via touch while performing a non-touch gesture that can be captured off-screen. The non-touch gesture can correspond to setting an action to be performed on the opponent, and because the non-touch gesture is performed and captured off-screen, it does not block the display or screen of the user device from the user's view. For example, the user device can detect touch input, such as a user's thumb touching an icon representing an opponent's chess piece in a chess game displayed on the screen of the user device. At the same time, the user device can detect gestures, such as a user closing the same hand (while touching an icon representing an opponent's chess piece) to remove the chess piece according to the rules of the game.

[0020] Timer: Select a timer area or icon via touch and gesture to adjust duration. Similar to the camera example described above, a user can select a timer area or icon representing a setting such as timer duration via touch while performing a non-touch gesture that can be captured off-screen. The non-touch gesture can correspond to adjusting the timer duration, and because it is performed and captured off-screen, it does not block the display or screen from the user's view. In one example, touching the clock and then rotating the hand one way or the other can also be used to set the time (e.g., moving forward / backward time can be determined by the number of extended fingers, which can be distinguished from changing hours or minutes). In another example, there may be only one way to adjust the time, such as a full rotation of the hand can adjust the minutes.

[0021] Maps: Select a point of interest (POI) via touch and gesture to adjust zoom, keeping the POI centered under the user's touch or hand. In one embodiment, the user can move their touch to adjust where the POI is displayed on the screen. Similar to the camera example described above, the user can select a POI on a map displayed on the user device via touch while performing a touchless gesture that can be captured off-screen. The touchless gesture can correspond to adjusting the zoom, and because it is performed off-screen and capture so it does not block the display or screen from the user's view.

[0022] Creating different types of "clicks": Selecting objects displayed on the screen of a user device via touch and gestures for clicking on the selected object (e.g., for initiating an action on the selected object). For example, a user may select an object such as a file displayed on the screen of a user device via touch while performing a non-touch gesture that may be captured off-screen. The non-touch gesture may include a closed hand that may be equivalent to a right mouse click (RMC), or an open hand that may be equivalent to a left mouse click (LMC) or an alternative click. In some embodiments, other hand gestures are used to represent RMC and / or LMC, or to represent different types of clicks.

[0023] Embodiments of the present invention may be applicable to many other use cases where compound commands may combine touch input and non-touch gestures for variable adjustment, including the following: The user device may detect touch input by a user touching an item with one hand (e.g., with a thumb while the hand is curled near the back of the user device to hold the device) and then sliding off the screen with the other hand. In one embodiment, for a system with "live tiles" (or other items with dynamic content), the sliding motion may scroll through the content (e.g., scrolling through email messages or scrolling up / down on a list). Conversely, this interaction may also work in another way, for example: the user device may detect the touch of an item, which may keep the item anchored in place, while the detection of a sliding gesture by the other hand may cause the "home screen" to switch behind the item. When the user has located the correct home screen on which to place the item, the user may release the item by removing his thumb from the screen to "throw" the item on the currently displayed home screen.

[0024] In various embodiments, the touch input may not necessarily be a static touch (or a single touch). Dragging an item from one place to another may have different effects (e.g., one finger extended vs. the entire hand open) based on the gesture (e.g., hand pose) detected when dragging. Similarly, pinch zooming may have different effects depending on the gesture detected, e.g., an open hand may achieve a coarse zoom, and a closed hand may achieve a fine zoom.

[0025] As described above, some use cases or instances may include adjusting one or more settings via a gesture using one or two hands. In some embodiments, there may be a difference in whether the gesture is performed with the same hand or a different hand. Thus, the user device may be configured in some embodiments to determine whether the same hand is touching the device and performing a gesture (e.g., a motion).

[0026] Advantageously, novel compound commands including both input targets and variable adjustments may be enabled, such as novel compound input gestures. Thus, the primary interface of the user device may be extended.

[0027] See also Figure 1 A-1C, Figures illustrate a process for enhancing touch using non-touch gestures according to an embodiment of the present invention. Specifically, Figure 1 A-1C illustrates an example for utilizing a touchless gesture to adjust the focus of a touch target on a camera.

[0028] exist Figure 1 In A, a user device 102, which may be a camera or have a camera as part of the user device, includes a screen or camera view finder for viewing a target item. In this case, the target item appears out of focus. The user may touch the screen of the user device 102 with an object such as a thumb at the point indicated by reference numeral 101 on the target item to be affected.

[0029] exist Figure 1 In B, the user performs an off-screen gesture to affect the target item. Figure 1 B, the user's finger is laterally offset from the device's display and can therefore be detected as performing an off-screen gesture. In one example, to focus on a target item, the user can rotate his or her hand in a plane generally parallel to the screen of the user device 102 to focus on the target item (as indicated by reference numeral 104) while maintaining his or her thumb at point 101 on the target item. The touchless gesture can be performed off-screen so that one or more sensors of the user device 102 (e.g., ultrasonic technology sensors or the like) can capture the off-screen gesture. The user can perform the rotating touchless gesture until the user determines that the target item is properly in focus. In one embodiment, the user can touch anywhere on the screen of the user device 102 to change the focus. The focus may be changed, or in other embodiments, the user may touch on a specific object to improve the autofocus.

[0030] In one embodiment, the user may touch a button, area, or icon that may be located near the edge of the camera (e.g., on the right edge or near the corner of the screen) so that the user can focus using a touchless gesture around the sides or corners of the display. In this way, the display is not blocked from view.

[0031] exist Figure 1 C, the user may release the touch from the user device 102 (eg, lift his or her thumb off point 101) to disengage from the target item.

[0032] In various embodiments, both gesture input sensing data and touch input sensing data may cease upon release of the touch from the user device, such as when the touch is released from a screen or display of the user device.

[0033] Thus, embodiments herein may provide a clean way to engage and disengage commands on a user device.

[0034] View now Figure 2 A-2B, a diagram illustrating a process for enhancing touch using non-touch screen gestures according to another embodiment of the present invention. Specifically, Figure 2 A-2B illustrates an example of a shortcut for refreshing or scrolling a list or page on a screen of a user device.

[0035] exist Figure 2 In A, the user may long press with a finger at a point 201 corresponding to a target item on the screen of the user device 202. Figure 2 In FIG. 2B , the user may perform a non-touch off-screen gesture, such as moving the hand downward (as indicated by reference numeral 207) while pivoting about the finger pressing the target item at point 201. In this manner, a list or page on the user device may be easily refreshed or scrolled, for example, without affecting other items on the screen. It should be noted that the non-touch gesture may be performed off-screen substantially along the same plane as the screen of the user device (i.e., in a substantially parallel plane), regardless of the proximity or distance of the user device from the moving hand in a perpendicular plane (i.e., in the z-direction).

[0036] In various embodiments, a particular touchless gesture may correspond to a particular command. Figure 2 In the example of A-2B, the downward movement of the hand, as indicated by reference numeral 207, may correspond to a "refresh" command, for example, for refreshing a page. In other embodiments, Figure 2 The combination of touch and non-touch gesture inputs illustrated in A-2B can be used as a shortcut to shuffle media.Such commands can be built-in, and in some embodiments, a lookup table can define which gestures correspond to which commands.

[0037] View now Figures 3A-3C , the diagram illustrates a process for enhancing touch using a non-touch gesture according to another embodiment of the present invention. Specifically, Figures 3A-3C An example is described for organizing items such as applications in a target destination such as a file.

[0038] exist Figure 3A , the user may long press with a finger at a point 301 corresponding to a target destination of one or more collected items on the screen of the user device 302.

[0039] exist Figure 3B , the user can begin selecting one or more items to be collected and placed at the target destination. In this regard, while long pressing with a finger at point 301, the user can use other fingers of the same hand to select the one or more items, such as item 302B. Alternatively, while long pressing with a finger at point 301, the user can use a finger from another hand to select the one or more items.

[0040] exist Figure 3C , once the user has completed selecting the one or more items to be collected, the user may perform a gesture such as a sweep gesture (e.g., as indicated by arrow 316) to move the selected one or more collected items toward a target destination indicated by long pressing the finger at point 301.

[0041] In this way, the one or more items are organized into the target destination, ie, the items are already organized into appropriate files or folders.

[0042] According to one or more embodiments of the present invention, various techniques may be used to enhance touch input with non-touch off-screen gestures. In this regard, any gesture detection techniques may be combined with touch input techniques. Such techniques may include, for example: on-screen hand touch plus ultrasonic gesture detection, on-screen hand touch plus vision-based gesture detection (e.g., using pre- and / or The invention can be performed by a user's touch screen, a rear-facing camera, and / or one or more cameras mounted on a side or edge of the device), on-screen ultrasonic pen touch plus hand vision detection (ultrasonic, vision-based, depth sensor, camera-based, IR, etc.), on-screen hand touch plus ultrasonic pen gesture detection, electromyography (EMG), and / or any other appropriate technology. In this way, gestures can be detected in front of the device, behind the device, and / or on one or more sides of the device. In some embodiments, the tip can be identified and / or tracked to determine the posture or movement. For example, the posture or movement of each finger can be identified; and a specific gesture model can be used. Touch screen technology can be used to detect touch input from the user, such as capacitive touch screens and / or projected capacitive technology. It should be noted that in various embodiments, touch on the screen may not be required. For example, touch input can be detected from a user touching the border or even just touching the edge of the device. For example, a user can rotate his hand, for example, to control the volume, which can be detected with, for example, a touch sensor located on the side, top, or bottom of the device or with an electric field sensor.

[0043] View now Figure 4, the flow chart illustrates a method for enhancing touch input using non-touch gestures according to an embodiment of the present invention. It should be noted that according to one embodiment, Figure 4 The method described in Figure 5 The system implementation described in .

[0044] In block 402, Figure 5 A system such as system 1500 described in the accompanying drawings can capture touch input data on the screen of a user device and non-touch gesture input data off the screen of the user device in parallel. Figure 5 The input component 1516 described in the above captures touch input data and non-touch gesture input data. In this regard, as described above with respect to including Figure 1 -3, a user can simultaneously touch a target item on the screen or display of a user device and use a non-touch gesture (with the same hand or another hand) to adjust the target item. For example, a user can select a target item on a camera, such as an image to be focused, by touching the image on the screen or viewer of the camera. At the same time, while touching the image on the screen of the camera, the user can perform an off-screen non-touch gesture, such as a hand motion or gesture, to adjust the focus of the image. In this regard, the off-screen non-touch gesture can be performed, for example, by a hand motion or gesture in a plane that is substantially parallel to the screen of the user device. And, the off-screen non-touch gesture can be captured by one or more sensors located on the user device, such as on the side, top, bottom, or back of the user device. The sensor can include various gesture detection technologies, such as ultrasonic technology, IR, or the like.

[0045] In block 404, the system may determine an input command based at least in part on the combination of the concurrently captured touch input data and non-touch gesture input data. In block 406, the operation of the user device may be affected based on the determined input command. This determination of the input command and its effect on the operation of the user device may be determined, for example, by Figure 5 In one embodiment, the input command may affect the selected target item without affecting any other unselected items.

[0046] In various embodiments, when the touch input data is released, the input command can be determined to be complete. Figure 1 In the camera focus embodiment illustrated in A-1C, once focus is achieved, the user can release the touch of the target item to disengage the target item.

[0047] According to one or more embodiments of the present invention, on-screen touch detection can enable off-screen object (e.g., finger) detection. The off-screen object detection can then be used to control the operation. For example, a rotating finger detected off-screen can be used to control the action of a setting or application. On-screen touch detection or touch input can be used, similar to an on-off switch, while non-touch gestures can be the operating controls. That is, the touch input can be the anchor point for non-touch off-screen gestures.

[0048] It should be noted that according to one or more embodiments of the present invention, the terms "gesture input sense data", "non-touch gestures" or "gestures" may generally include off-screen non-touch gestures or interactions (other than touch screen interactions) captured by user sensors of a device, and may include, for example, non-touch hand motions or postures. Thus, in embodiments herein, non-touch gestures may include detecting or recognizing interactive input from a user such as hand gestures. In various embodiments, "gesture input sense data" may include using data from at least one or more points or areas (e.g., sides, top, bottom, back, etc.) of a device (other than touch sensors). The user actions may be interpreted as control inputs using data from sensors at the user's device or from interactions other than those performed on or directly over the touch screen. That is, gesture input data may involve capturing a user's off-screen gestures using sensors or technology without touching the user device.

[0049] In general, a user device may include or be part of any device that includes gesture recognition or gesture input data sensors, which may include, for example, ultrasonic gesture detection sensors, image or video capture components such as cameras (e.g., visible light cameras, range imaging cameras such as time-of-flight cameras, structured light cameras, stereo cameras, etc.), hand vision detection (ultrasonic, vision-based, IR, etc.) sensors, ultrasonic pen gesture detection sensors, etc. That is, the device may have ultrasonic, IR, vision-based gesture capabilities, etc. to capture the user's off-screen gestures without touching the device (i.e., non-touch gestures such as hand gestures).

[0050] In various embodiments, touch inputs may affect the operation of other sensors. For example, touch inputs may, for example, turn other sensors on or increase the duty cycle of such other devices. Also, touch inputs may cause adjustments to gestures that other sensors are configured to detect. For example, instead of being sensitive to an engagement gesture to initiate gesture control, touch inputs may alert the device to look for gestures such as swipes, swipes, gestures, etc. other than the engagement gesture. In some embodiments, this may be accomplished by loading one or more gesture models, increasing the size or scope of the gesture dictionary, and / or in any other number of ways.

[0051] With respect to capturing off-screen non-touch gestures according to one or more embodiments, the movement of a user or the orientation of an object (e.g., one or more fingers, hands, pens, etc.) beyond the surface of a user device (e.g., beyond the screen) may be substantially parallel to the surface of the user device, such as a screen or display. In various examples, the movement of an object parallel to the surface of the device may include sweeping motions, thumb rotations, etc. beyond the boundaries of the screen or display on the same plane as the display or screen of the device, and while movement in the z-direction may be considered in some embodiments, it may not be the only movement considered or may not be considered in some embodiments. In this regard, z-direction movement may, for example, be movement in a direction toward or away from the screen or display, such as to increase or decrease proximity to the display. That is, the movement of an object may be performed outside the perimeter of the screen and aligned with the screen. In some embodiments, a gesture may be detected when aligned with the screen (e.g., when using a tablet computer or larger form factor).

[0052] View now Figure 5 , illustrating a block diagram of a system for implementing an apparatus according to an embodiment of the present invention.

[0053] It should be understood that the methods and systems disclosed herein can be implemented by incorporating into a wide variety of electronic systems or devices. For example, the system 1500 can be used to implement any type of device including a wired or wireless device, such as a mobile device, a smart phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, a personal computer, a TV, or the like. Other exemplary electronic systems such as music players, video players, communication devices, network servers, etc. can also be configured according to the present invention. In addition, other electronic systems may include, for example, a wearable device mounted on a user's wrist that can detect gestures or other control inputs and then pass them to another device for processing.

[0054] System 1500 may be suitable for implementing embodiments of the present invention, including Figure 1 , 2 and the user device described in 3 102, 202, and 302. System 1500, e.g., a portion of a device such as a smart phone, tablet computer, personal computer, and / or network server, includes a bus 1502 or other communication mechanism for transmitting information, which interconnects subsystems and components, including one or more of the following: a processing component 1504 (e.g., a processor, a microcontroller, a digital signal processor (DSP), etc.), a system memory component 1506 (e.g., RAM), a static storage component 1508 (e.g., ROM), a network interface component 1512, a display component 1514 (or, an interface to an external display), an input component 1516 (e.g., a keypad or keyboard, an interactive input component (e.g., a touch screen), gesture recognition, etc.), and a cursor control component 1518 (e.g., a mouse pad). In various embodiments, input component 1516 may implement one or more gesture detection techniques that may be combined with touch input techniques. As shown in FIG. As described above, such technologies may include, for example, hand touch on the screen plus ultrasonic gesture detection, hand touch on the screen plus vision-based gesture detection (e.g., using front and / or rear-facing cameras and / or one or more cameras mounted on a side or edge of the device), ultrasonic pen touch on the screen plus hand vision detection (ultrasonic, vision-based, depth sensor, camera-based, IR, etc.), hand touch on the screen plus ultrasonic pen gesture detection, electromyography (EMG), projected capacitance, electric field sensor, and / or any other appropriate technology.

[0055] According to an embodiment of the invention, system 1500 performs specific operations by executing one or more sequences of one or more instructions contained in system memory component 1506 by processing component 1504. Such instructions may be read into system memory component 1506 from another computer-readable medium (e.g., static storage component 1508). These may include instructions to enhance interactive input via simultaneous multi-modal sensor fusion. In other embodiments, hard-wired circuits may be used in place of or in combination with software instructions for implementing one or more embodiments of the invention.

[0056] Logic may be encoded in a computer-readable medium, which may refer to any medium that participates in providing instructions to processing component 1504 for execution. This medium may be in many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. In various implementations, volatile media include dynamic memory (e.g., system memory component 1506), and transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 1502. In an embodiment, transmission media may be in the form of sound waves or light waves, such as those generated during radio wave and infrared data communications. Some common forms of computer-readable media include, for example, RAM, PROM, EPROM, flash-EPROM, any other memory chip or cartridge, carrier wave, or any other medium from which a computer is suitable for reading. The computer-readable medium may be non-transitory.

[0057] In various embodiments of the present invention, execution of the instruction sequence for practicing the present invention may be performed by system 1500. In various other embodiments, multiple systems 1500 coupled by communication link 1520 (e.g., WiFi or various other wired or wireless networks) may coordinate execution of the instruction sequence for practicing the present invention. System 1500 may receive and expand input, messages, data, information, and instructions via communication link 1520 and network interface component 1512, including one or more programs (i.e., application code). The received program code may be executed by processing component 1504, which receives and / or stores in disk drive component 1510 or some other non-volatile storage component for execution.

[0058] As those skilled in the art will now appreciate and depending on the particular application at hand, many modifications, substitutions and changes may be made in the materials, equipment, configurations and methods of use of the devices of the present invention without departing from the spirit and scope of the invention. In view of this, the scope of the present invention should not be limited to the scope of the specific embodiments illustrated and described herein, as they are merely by way of some examples thereof, and in fact should be fully commensurate with the scope of the appended claims and their functional equivalents.

Claims

1. A method for enhancing digital device input using gestures, comprising: capturing touch input data on a screen of a user device, the touch input data indicating a touch performed on the screen of the user device; determining an input command based on the touch input data; affecting an operation of the user device based on the touch input data, wherein the operation affects display of content on the screen of the user device; capturing touchless gesture input data indicative of a gesture performed in an area offset from the screen of the user device, the touchless gesture input data being associated with the determined input command; and further affecting the operation of the user device based on the determined input command and the touchless gesture input data.

2. The method of claim 1, wherein the touch input data identifies a target item on the screen of the user device, and wherein the input command comprises adjusting an aspect of the target item based on the non-touch gesture input data. 3 . The method of claim 2 , wherein the input command comprises a variable adjustment of the target item, the variable adjustment being determined from the touchless gesture input data.

4. The method according to claim 1, wherein the capturing of the touch input data further includes receiving a touch from a user on the screen of the user device to affect a desired target item; and determining that the target item has been detached by detecting the release of the touch on the screen of the user device. The method of claim 1 , wherein said capturing said touchless gesture input data comprises detecting a position and movement of an object. The method of claim 5 , wherein the movement of the object further comprises movement substantially in the same plane as the screen of the user device.

7. The method of claim 1, wherein the capturing the touchless gesture input data comprises using one or more sensors adapted to detect objects beyond a surface of the user device via ultrasonic technology, image or video capture technology, or IR technology. 8 . The method of claim 1 , wherein the determined input command comprises one of a plurality of different types of inputs indicative of selection of a target item on the screen of the user device.

9. The method of claim 8, wherein the different types of input further include a right mouse click (RMC) generated by a first posture of the hand touching the screen, or a left mouse click (LMC) or an alternative click generated by a second posture of the hand touching the screen, the first posture being different from the second posture.

10. The method of claim 1, wherein the capturing the touchless gesture input data comprises capturing a hand pose or a hand motion.

Images