Applications for multi-touch input detection

Through the user's wearable device to identify idiotic and non-idiopic hands, the problem of multi-touch command interpretation errors on large touch screens is solved, the processing efficiency and user experience are improved, and the convenience of multi-person collaboration is enhanced.

CN113608665BActive Publication Date: 2024-11-29MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202110907417.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2017-03-23
Publication Date
2024-11-29
Estimated Expiration
2037-03-23

AI Technical Summary

Technical Problem

Existing multi-touch input detection technology is difficult to accurately interpret multiple or simultaneous touch commands on large touch screens, resulting in input errors and low processing efficiency, and users need to move more steps when manipulating data, affecting user experience and collaboration efficiency. .

Method used

The user wearable device (UWD) is used in combination with motion sensors to distinguish multi-touch events by identifying the user's idiomatic and non-idiomatic hands, and provide personalized menus and tactile feedback on the display, improving interactivity and collaboration efficiency.

Benefits of technology

It effectively avoids input errors on the touch screen, improves the processing efficiency and user experience of computing devices, reduces the operation steps and time of users on large monitors, and enhances the convenience of multi-person collaboration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113608665B_ABST
    Figure CN113608665B_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure relate to applications of multi-touch input detection. Typically, the multi-touch detection implementations described herein use touch detection technology to provide new and advantageous interactions between users and touch displays using touch, pen, and user wearable devices (UWD). These new and advantageous interactions include user-associated mobile menus, combined click and radial marking menus, menus for automating and improving the drawing or manipulation of content on displays, new menus and methods for selecting objects and text on displays, and new interactions with UWD and touch screen displays using gestures using UWD. In addition, targeted tactile feedback of the UWD of a specific user of the display is enabled. In some multi-touch detection implementations, menus or tools that can be used to act on objects on the display can be transplanted completely or partially between displays, such as between a small display and a large display.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of an invention patent application whose international application date is March 23, 2017, entered the Chinese national stage on September 27, 2018, has a Chinese national application number of 201780020764.9, and the invention name is “Application of multi-touch input detection”. Technical Field

[0002] Embodiments of the present disclosure relate to applications for multi-touch input detection. Background Art

[0003] The popularity of smart phones, tablet computers, and many types of information devices is driving the demand and acceptance of touch screens for various types of electronic devices. Touch screens are often used in situations where keyboard and mouse systems do not allow the user to interact with the contents of the display appropriately intuitively, quickly, or accurately. Many mobile computing devices (e.g., tablet computers, smart phones, etc.) have touch screens and use a pen, pointer, or pen-type input device in conjunction with a digitizer component of the computing device for input purposes. Many of these mobile computing devices allow interaction with these touch-sensitive screens using a pen and bare hand touch or a combination of both.

[0004] In addition, the size of touch screen displays continues to increase. For example, large touch screens (e.g., greater than eighty inches diagonally) are used as presentation tools in front of an audience or for collaboration between staff members. These touch screens allow a user to manipulate an object displayed by the touch screen (e.g., by selecting the object, positioning it (e.g., via dragging), etc.). Summary of the invention

[0005] This Summary is provided to introduce some concepts in a simplified form, which are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0006] In general, the multi-touch detection implementation described herein uses touch detection technology to provide new and advantageous interactions between users and touch displays using touch, pen, and user wearable devices (UWD). These new and advantageous interactions include user-associated mobile menus, combined click and radial marking menus, menus for automating and improving the drawing or manipulation of content on the display, new menus and methods for selecting objects and text on the display, and new interactions with UWD and touch screen displays by using gestures to use UWD for manipulating menus and objects displayed on the display. In addition, targeted tactile feedback of the UWD of a specific user of the display is enabled. In some multi-touch detection implementations, menus or tools that can be used to act on objects on a large screen display can be fully or partially ported to a small screen device.

[0007] Some multi-touch detection implementations described herein relate to interactive touch screen displays, particularly interactive touch screen displays capable of distinguishing and responding to multiple simultaneous touches (multi-touch) and / or one or more digital styluses (pen and / or touch). While some implementations may be applied to larger interactive digital whiteboard-type displays, other implementations may be applied, for example, to smaller versions of interactive displays (such as those incorporated in so-called board-type or tablet computers) and even smaller devices (such as touch-screen operated smartphones). In some multi-touch detection implementations described herein, menus and tools may be split or distributed between a small display and a large display.

[0008] The multi-touch detection implementation described in this article has many advantages. For example, when using a large touch screen display, if multiple touch commands are received, the processor of the computing device / system can correctly interpret the input. This improves computing efficiency and usability by avoiding errors from multiple or simultaneous touch commands on a display (e.g., a large touch screen) and errors in the interpretation of unintentional input. In addition, since many multi-touch detection implementations described in this article provide menus that move with the user's hand or finger relative to a large display, the user must reach less and move less to manipulate the data on the large display compared to the case where one or more menus are fixed at a large display. In some multi-touch detection implementations, the outlines of some of these menus can also be used as drawing tools without calling out a specific drawing tool menu or control. By wearing a UWD or other device associated with a specific user that allows identification of each user, the user experience is improved by allowing personalized menus and capabilities for processing information and objects on the screen, including, for example, personalized menus based on the user's past actions or preferences, access to personal data (e.g., images, documents or other files), and personalized tactile feedback to one or more specific users, including personalized tactile feedback to the UWD of a specific user. Furthermore, since multiple persons simultaneously manipulating objects and information displayed on the display using both pen and touch can be distinguished, the time to complete various tasks on the display can be reduced and collaboration between users can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The specific features, aspects and advantages of the present disclosure will be better understood with respect to the following description, appended claims and accompanying drawings, in which:

[0010] Figure 1 is a block diagram depicting an exemplary computing environment in which a multi-touch detection implementation as described herein may be implemented.

[0011] Figure 2 is a schematic diagram of an exemplary system including a touch screen and a user of the touch screen according to an exemplary multi-touch detection implementation as described herein.

[0012] Figure 3 Included is a timing diagram of example touch events performed by the arm / hand wearing the user's wearable device.

[0013] Figure 4 A user's arm / hand is depicted interacting with exemplary objects displayed on the touch screen.

[0014] Figure 5 Two users' arms / hands are depicted interacting with an object displayed on a touch screen.

[0015] Figure 6Depicted is an exemplary combined tap and radial marking menu for modifying stroke characteristics of a stroke drawn on a touch screen display.

[0016] Figure 7 Depicted in Figure 6 Modified stroke properties of a stroke drawn on a touch screen display depicted in FIG.

[0017] Figure 8 Depicted is the use of two fingers or a finger and thumb to select content on a touch screen through an associated display menu referred to herein as an x-ray menu. The selection of content can be redirected by repositioning, resizing, or rotating the two fingers (or finger and thumb) or some other predetermined number of fingers / thumbs. One or more other body parts, pen(s), and / or other object(s) may also be used to create an x-ray menu. This example shows the selection of the handwritten text "slope" and the display of an associated menu for applying text recognition to the handwritten text.

[0018] Fig. 9 Depicted Figure 8 An expansion of the text recognition menu shown in . Menu 902 can also be selected using a stylus (or other object) instead of a finger.

[0019] Fig.10 Depicted through Figure 8 and 9 The combined click radial marking menu shown in is activated to convert the handwritten text "slope" into type written "slope" after applying text recognition. In some implementations, once the finger is removed from the display, the X-ray menu is removed and the radial marking menu is also invisible.

[0020] Fig.11 A menu is depicted whose outline can be used as a drawing tool, such as a curved board, a straight edge, a circle, or other shapes that can be used to help the user draw. The outline of the menu can be changed by the user and can also be used to perform other tasks besides drawing (e.g., cutting or masking content on the display) without having to activate dedicated controls for these purposes.

[0021] Fig.12 Depicted Fig.11 Modification of a curve drawn in , where a user modifies the curve by modifying characteristics of the menu outline using touch or multi-touch (ie, with the user's dominant hand).

[0022] Fig.13 A user is depicted using a handheld computing device with a small screen to implement a multi-touch detection implementation as described herein.

[0023] Fig.14Depicted are exemplary computer-implemented processes that may be used to practice various multi-touch detection implementations as described herein.

[0024] Fig.15 Depicted is an exemplary expansion of a combined point-and-click and radial marking menu.

[0025] Fig.16 Depicted are exemplary menus accessible via a combined click and expansion of a radial marking menu employed in some multi-touch detection implementations described herein.

[0026] Fig.17 Depicting a combined tap and radial menu from which a particular user's photos may be selected. DETAILED DESCRIPTION

[0027] In the following description of multi-touch detection implementations as described herein, reference is made to the accompanying drawings, which form a part of the description and show by way of illustration examples in which the implementations described herein may be practiced. It should be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the claimed subject matter.

[0028] 1.0 Multi-touch detection implementation

[0029] The following sections provide an overview of multi-touch detection implementations, a description of the underlying technologies for implementing the multi-touch detection techniques described herein, and a description of an exemplary computing environment. Systems in which the multi-touch detection implementations described herein can be implemented and various exemplary multi-touch detection implementations may also be provided.

[0030] As a preliminary matter, some of the following figures describe concepts in the context of one or more structural components, which are variously referred to as functions, modules, features, elements, etc. The various components shown in the figures can be implemented in any way. In one case, the separation of the various components shown in the figures into different units can reflect the use of corresponding different components in the actual implementation. Alternatively or additionally, any single component shown in the figures can be implemented by multiple actual components. Alternatively or additionally, the depiction of any two or more separated components in the figures can reflect the different functions performed by a single actual component.

[0031] Other figures describe concepts in the form of flow charts. In this form, certain operations are described as constituting different blocks that are executed in a particular order. These implementations are illustrative and not restrictive. Certain blocks described herein may be combined together and executed in a single operation, certain blocks may be divided into multiple component blocks, and certain blocks may be executed in an order different from the order shown in this article (including a parallel manner of executing the blocks). The blocks shown in the flow charts may be implemented in any manner.

[0032] 1.1 Overview

[0033] In general, the multi-touch detection implementations described herein use multi-touch detection on a display (or other touch-sensitive device) (e.g., touch by one or more fingers, one or more pens, and / or one or more other objects) to provide new and advantageous interactions between a user and a touch display or other touch-sensitive device using touch, a pen, a user wearable device (UWD), and / or other objects or devices (some of which may be associated with a given user). These new and advantageous interactions include user-associated mobile menus, combined click and radial marking menus, menus that automate and improve the drawing or manipulation of content on a display using the outline of a menu, new methods for selecting and modifying objects and text on a display, and new interactions with UWDs and touch screen displays using gestures, etc. In addition, targeted tactile feedback to the UWDs of one or more specific users of the display is enabled. The various implementations described herein can be implemented with a pen, finger / hand, or other object that touches a display.

[0034] 1.2 Multi-touch detection relative to the display using a user wearable device

[0035] This section provides an overview of exemplary techniques by which a processor of a computing device can identify and classify multiple touches to a display by one or more users' dominant and non-dominant hands in order to improve usability of large or small displays by using various implementations as described herein. This technique, as well as other techniques capable of discerning touches on a touch screen display and identifying which user and which hand the user is touching the display with, can also be used to implement the various multi-touch detection implementations described herein.

[0036] When interacting with an object or device, each hand of the user (defined herein as a person with a right and left hand or with one hand and an opposite hand) can have a different role. For example, the non-dominant hand can be used exclusively to locate the workspace of the dominant hand. In some configurations, the device can distinguish between the dominant hand and the non-dominant hand of the user by receiving a wireless signal representing the movement of a device (e.g., a band or ring) worn by the user. The device referred to herein as a user wearable device (UWD) may include, for example, a motion sensor, such as an accelerometer and / or a gyroscope or an inertial measurement unit (IMU) type. The technology involved can associate the movement of the UWD with (multiple) touch events of the touch screen (and detected by the touch screen). In some implementations, such technology can allow detection (or determination) of which user (and which hand of the user) of two or more users is performing a touch event using a touch screen (or "display surface", which can include any surface that can include a displayed image). However, it should be noted that other non-wearable devices can perform the functions of the UWD described herein. These can be, for example, handheld devices, or sensor transmitter pairs, or some other implementation that can give the same or similar effects as those provided by the UWD as described herein.

[0037] Various terms herein can be used for touch display devices, such as touch screens, touch displays, etc. Unless otherwise specified in a specific context, these terms are substantially equivalent. A touch screen may include an input device that is usually layered on top of an electronic visual display of an information processing system. By using a special stylus / pen, one or more fingers, one or more hands, or other body parts to touch the display, a user can use a simple or multi-touch gesture to provide input or control an information processing system during a touch event. For example, a user can use a touch display to react to displayed content and control how it is displayed (e.g., by extending (zooming) text size, selecting menu items or objects, etc.). Herein, a touch event may involve a user (e.g., a user's (multiple) fingers, (multiple) hands) or a physical touch between an object such as a stylus and a touch screen, or may involve a user (e.g., a user's (multiple) fingers, (multiple) hands) or an object such as a stylus that is relatively close to (e.g., a few millimeters or a few centimeters) the surface of the touch screen without touching the touch screen. In some examples, a touch event may refer to a press event that exceeds a specific pressure level (force) on the contact surface, such as the pressure level required for a resistive touch screen. Unless otherwise indicated, the term "touch event" refers to a touch event, a hover event, or a combination thereof. As used herein, the term "arm / hand" is used to refer to any part of a user's arm or hand, from the shoulder to any of the fingers or thumb. For some examples, a touch event for an arm / hand may involve one or more fingers touching the touch screen, the side of a hand touching, hovering, or passing over a portion of the touch screen, or a specifically oriented forearm above the touch screen. The claimed subject matter is not so limited.

[0038] A touch screen can enable a user to interact directly with displayed display objects (e.g., windows, menus, text, pictures, icons, images, etc.) rather than using a mouse, touchpad, or any other intermediate device. Although some examples herein describe a "screen," the techniques herein can be similarly applied to touch surfaces that do not have an integrated display or that have a display positioned separately from an input surface (or a display projected solely on an input surface). For example, UWD sensing technology can be used in conjunction with a grip sensor, as well as in conjunction with a touch screen (e.g., to sense which hand and / or user is in contact with the rear surface of a grip-sensing tablet). Therefore, in some examples, the combination of a touch-sensitive surface plus a UWD does not need to involve a display.

[0039] Touch screens can be used in devices such as game consoles, personal computers, tablet computers, smart phones, large display screens (e.g., located at the front of a classroom or lecture hall), etc. Touch screens can be attached to (multiple) computers or use client devices (e.g., as terminals) for use in a network. Touch screens can be integrated in the design of digital devices such as personal digital assistants (PDAs), GPS navigation devices, mobile phones, video games, e-books, etc.

[0040] Various examples describe techniques and architectures for systems that can, among other things, detect the dynamics of an arm / hand touching a touchscreen or non-touch-sensitive surface or display. For example, the system can detect or determine the impact (e.g., velocity and acceleration in three dimensions) of an arm / hand touching a display or other surface.

[0041] In various examples, for a user wearing one or more UWDs that stream motion sensing data to a receiver associated with a touch screen, the system can allow portability and distributed processing and display of such technology on any of a variety of types of touch screens such as smartphones, tablets, large displays, etc.

[0042] In some configurations, the system can automatically detect, identify, or distinguish multiple users operating (or approaching) a touch screen. This may be useful, for example, in situations where multiple people work side by side and / or collaborate on a large touch screen. In some implementations, cluster information can be used to detect a user's actions even if the user's hands are not wearing a UWD. In some implementations, menus individually customized for specific users can follow individual users across multiple devices. In some implementations, for example, a finger clipboard can be used to carry files through the cloud.

[0043] Some configurations described herein relate to UWDs, which may be wristbands, bracelets, patches (e.g., adhesive patches), gloves (e.g., which need not cover the entire hand), etc. In some cases, a UWD may be a watch or considered jewelry. Other examples of UWDs include bands (e.g., bracelets, clasps, bracelets, etc.), rings (e.g., one or more on (multiple) fingers or (multiple) thumbs), armbands or forearm cuffs or other electronic textiles (e.g., electronic clothing), elbow pads or armbands worn on the upper arm, mobile devices (e.g., phones) fixed or strapped to a limb, bandages, electronic tattoos with embedded sensors, electronic skin or implants, subcutaneous sensors (e.g., relatively small modules that can be implanted or injected under the skin), electronic nails (e.g., stickers with display and / or sensing capabilities), or sensors fixed as part of piercings or other jewelry, etc.

[0044] In any case, the UWD may be configured to wirelessly transmit an electronic signal representing the motion (e.g., translation, rotation, orientation, rate, speed, acceleration, etc.) of the UWD and / or an identification (ID) of the wearer of the UWD. Such an ID may include identification information of a specific user or a group of users. For example, an identification indicating that the user is a member of the faculty rather than a student may allow access to specific user privileges or data of a faculty user that is not provided to a student. For example, the UWD may include a memory device, a transmitter and / or a transceiver, and / or a processor, but the claimed subject matter is not limited thereto. In some implementations, a user may wear more than one UWD (e.g., a band on a non-dominant hand and a ring on a dominant hand). In some implementations, the UWD may also wirelessly receive an electronic signal from a computing device that may control a display or other device that the user is using. For example, the UWD may receive tactile feedback for an action taken by the wearer of the UWD on an object displayed on a display. The tactile feedback may be at the same time or after a touch event, or may not be associated with a touch event at all. In some implementations, the UWD may be configured to wirelessly receive electronic signals representing a command to cause the UWD to perform a certain function, such as vibrating or providing an audible alarm in response to the command.

[0045] In some examples, the ID can be in the form of a globally unique identifier (GUID) or a MAC address or other unique string derived from the connection (pairing) of the UWD to a display or other proximity device. In some implementations, the user's username and / or user information (e.g., user preferences) can be found on the Internet or in the cloud by using the UWD's GUID (e.g., a username not stored in the UWD). In some specific examples, detecting or establishing the ID does not require any additional hardware other than the sensor and wireless connection.

[0046] In some configurations, when a user's finger touches the touch screen, a "touch down" event (sometimes called a "pointer press") may be generated through an application programming interface (API). The event may be in response to a finger touching the touch screen. In some configurations, the event may involve information that may allow the processor to determine, for example, which hand of the user or which user of multiple users touched the touch screen. For example, the information may include identification data of the user. The data may be stored in a UWD worn by the user. The data may be wirelessly transmitted to a processor (e.g., a processor of the touch screen) in the following situations: (i) after or in parallel with a touch down event, (ii) from time to time or periodically, (iii) when the user (and UWD) is within a certain distance from the touch screen, or (iv) at the beginning of a period of use of the touch screen, to name just a few examples.

[0047] A touch down event may involve information that may allow a processor to determine the strength of the touch. As described below, such a determination may be based at least in part on motion detected using an inertial measurement device such as an accelerometer, gyroscope, compass, or the like.

[0048] In some example configurations, the actions of the processor and the associated actions of the touch screen (e.g., what object or image can be displayed after an event) can be based at least in part on the determination of: (i) which hand of the user, or which user of multiple users, touched the touch screen, (ii) the orientation of the hand at the time of the touch, (iii) the orientation of the hands of other users at the time of the touch, and (iv) the intensity of the touch, etc. For example, if it is determined that a specific user has touched the touch screen, a menu customized for the specific user can be displayed (e.g., a priori or based on the commands and tools recently used by the specific user on the current device or elsewhere). In another example, if it is determined that the user's dominant hand has touched the touch screen, the touch screen can therefore display different objects or images than if it is determined that the non-dominant hand has touched the touch screen. In yet another example, if it is determined (or detected) that the intensity is higher than a certain threshold (e.g., touch impact-how hard the user touches the screen), the touch screen can ultimately display different objects or images than if it is determined (or detected) that the intensity is lower than a certain threshold. In another example, there may be more than one threshold.

[0049] As used herein, the phrase "modifies at least one object displayed by the touch screen" means that the touch screen (e.g., or other type of display or surface) changes what it displays (e.g., a window, menu, icon, graphical object, text, etc.) or how it displays an object or display background (e.g., the brightness and / or contrast of a particular portion of the touch screen). In some examples, the system can use indirect input (e.g., as in desktop computing, where a device on a desktop triggers an action on a separate display). In other examples, the system can use (multiple) interactions on a projection surface. The surface itself need not be a display, such as when an image is projected onto a surface such as a wall or table.

[0050] A system that may include a touch screen, a processor of the touch screen, and a UWD worn by a user may receive signals wirelessly from the UWD. Signals representing the movement of the UWD (and therefore the associated movement of the user) may include shapes, contours, spikes, and the like. For example, a signal representing acceleration plotted as a function of time may include acceleration pulses (e.g., spikes or sharp contour pulses) that may indicate a downward touch event. This is because peak acceleration may occur when a user's finger or hand touches a surface (e.g., a touch screen). Therefore, a pulse of the acceleration signal may indicate such a downward touch event. Such a pulse may have a pulse width of approximately 100 milliseconds or less (e.g., full width at half maximum (FWHM)), but the claimed subject matter is not limited thereto.

[0051] In some configurations, if the width of the acceleration pulse is below a (predetermined) threshold and its height (amplitude) is above a detection threshold, the system's processor can determine that there is a correlation between a downward touch event and an acceleration pulse generated by the UWD. Therefore, the hand wearing the UWD (the non-dominant hand may wear the UWD, but this is not necessarily the case) may have just touched the touch screen (or other surface). The system's processor (e.g., via an API) can then initiate a hand detection event, which will have information about which hand just touched (e.g., in this case, the non-dominant hand), user information (because the API "knows" which UWD movement produced the accelerometer spike at this time), and the intensity of the touch (based on the shape of the spike (pulse)). In some examples, the API can return a contact ID and a finger position (e.g., which can be sensed by a standard touch API). For illustrative purposes, input provided by the non-dominant hand is referred to herein as non-dominant input, and input provided by the dominant hand is referred to herein as dominant input.

[0052] As just described, in some configurations, a UWD worn by a hand performing a touch can generate acceleration pulses and / or other changes in motion rate. The UWD worn on the hand performing the touch can measure an acceleration profile when a touch occurs. (In some implementations, the UWD can stream the transmitted sensor data while the processor controlling the display measures the acceleration profile or other motion parameters based at least in part on the sensor data. Similarly, in some implementations, the controller on the touch display can notify the UWD when a touch event is received, and then the UWD can notify the display controller when a spike in the acceleration and other motion parameters of the UWD is measured). On the other hand, a hand that is not wearing a UWD can instead touch. Even if the hand that is not wearing the UWD touches and the hand that is wearing the UWD does not touch, the UWD can still generate an acceleration profile. This is because the movement of one part of the user can be transferred (e.g., a "vibration" wave, weight transfer, etc.) to another part of the user. The difference is as follows: the UWD of the hand performing the touch can generate a relatively peak profile with a relatively large amplitude. But the UWD worn on the other hand (which does not perform the touch) can generate a relatively flat pulse in some implementations

[0053] In some configurations, a user may wear multiple UWDs on the same arm (i.e., ring and band). In this case, for example, the system may be able to determine the orientation of the fingers and wrist to create more complex gestures. In this case, the motion signals provided by each device may also be combined to more confidently or more sensitively detect spikes in the motion signals.

[0054] In some configurations, the UWD can be worn on a non-dominant hand, but the system can be automatically notified of the location of the wearing UWD based at least in part on user preferences for a particular user. Such user preferences can be provided from a memory located in the cloud or part of the system, entered by the user, or can be collected directly from the UWD (e.g., user settings stored on the UWD). In the case of incomplete knowledge, default settings can also be used (e.g., the user typically wears a watch on the left hand). In another instance, machine learning and / or cameras, etc. can be used to determine the location of the wearing UWD. In another instance, these methods can also be mixed to determine the location of the wearing UWD. The configuration described herein can be applied to a tablet (e.g., not limited to large displays and smart phones) or any type of device that can be touched (with or without a display).

[0055] In the examples herein, although elements such as user, object, finger, hand, UWD, processor, etc. may be stated in the singular, the claimed subject matter is not limited thereto. Thus, for example, more than one such element may be implied unless otherwise specified.

[0056] 1.3 Exemplary Operating Environment

[0057] The environment described below constitutes one exemplary operating environment for practicing various multi-touch detection implementations as described herein, and is not intended to limit the claims to any one particular operating environment. Other environments may be used without departing from the spirit and scope of the claimed subject matter.

[0058] Figure 1 An exemplary environment 100 in which the exemplary processes described herein may operate is shown. In some examples, various devices and / or components of environment 100 include various computing devices 102. As an example and not limitation, computing device 102 may include devices 102a-102f that may be interconnected via network 103, which may include, for example, the Internet or a cloud (e.g., a computing cloud). Although shown as a variety of device types, computing device 102 may be other device types and is not limited to the device types shown. Computing device 102 may include any type of device having one or more processors 104 operably connected to input / output interface 106 and memory 108, for example, via bus 110. Computing device 102 may include a personal computer, such as a desktop computer 102a, a laptop computer 102b, a tablet computer 102c, a telecommunication device 102d, a personal digital assistant (PDA) 102e, (multiple) touch screens 102f, an e-book reader, a wearable computer, an automotive computer, a gaming device, a measuring device, and the like. Computing device 102 may also include business or retail oriented devices, such as server computers, thin clients, terminals, and / or workstations. In some examples, computing device 102 may include components, such as for integration into a computing device, an apparatus, or other type of device.

[0059] In some examples, some or all of the functions described as being performed by computing device 102 may be implemented by one or more remote peer computing devices, one or more remote servers, or cloud computing resources. In some examples, computing device 102 may include one or more processors 104 to receive kinematics and / or user identification from a UWD via input / output 106, which may include, for example, a wireless receiver.

[0060] In some examples, as shown with respect to the touch screen 102f, the memory 108 can store instructions executable by the processor 104 (including an operating system (OS) 112) and programs or applications 116 loadable and executable by the processor 104. The one or more processors 104 may include one or more central processing units (CPUs), graphics processing units (GPUs), video buffer processors, etc. In some implementations, the machine learning module 114 includes executable code stored in the memory 108 and executable by the processor 104 to collect information locally or remotely via the input / output 106 through the computing device 102. The information may be associated with one or more applications 116. The machine learning module 114 may selectively apply any one of a plurality of machine learning decision models stored in the memory 108 (or more specifically, stored in the machine learning 114) to apply to the input data. For example, machine learning may involve a process that involves the processor interpreting or determining a user action based at least in part on information received from the UWD 118, which may communicate wirelessly with a device (e.g., 102) operating the touch screen 102f via a wireless communication path 120. In some implementations, the touch screen 102f can include a display surface (e.g., any surface such as a wall, table, etc.) associated with the processor 104. For example, the touch screen 102f can be a wall on which an image is displayed via a projector operated by the processor.

[0061] Although certain modules have been described as performing various operations, modules are merely examples, and the same or similar functions may be performed by a greater or lesser number of modules. In addition, the functions performed by the described modules do not necessarily have to be performed locally by a single device. Instead, some operations may be performed by a remote device (e.g., peer, server, cloud, etc.).

[0062] Alternatively or additionally, some or all of the functions described herein may be performed at least in part by one or more hardware logic components. For example, and not limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), and the like.

[0063] In some examples, computing device 102 may be associated with a depth camera that can be used to measure the distance from the camera to various parts of the image captured by the camera. In some cases, individual pixels of an image may have associated distance data that specifies the distance from the camera to the portion of the image corresponding to the corresponding pixel. In some examples, computing device 102 may be associated with a camera capable of capturing images and / or video and / or a microphone capable of capturing audio. For example, input / output module 106 may include such a camera and / or microphone. For example, a captured image of a user of a touch screen may be compared with an image in a database of users stored in memory 108, and such a comparison may be used in part to identify the user. Voice audio may be compared with audio in a database of users stored in memory 108, and such a comparison may be used in part to identify the user. Such identification may be used in conjunction with identification information provided by a UWD worn by one or more users. Memory 108 may include one or a combination of computer-readable media.

[0064] Computer-readable media may include computer storage media and / or communication media. Computer storage media include volatile and nonvolatile removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computing device.

[0065] In contrast, communication media contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave and / or other transmission mechanism. As defined herein, computer storage media do not include communication media. In various examples, memory 108 is an example of a computer storage medium storing computer executable instructions. When executed by processor 104, the computer executable instructions configure the processor to receive kinematic data of the UWD, particularly during a touch event of the touch screen performed by a user; and modify at least one object displayed by the touch screen based at least in part on the received kinematic data.

[0066] In various examples, the input device of the input / output (I / O) interface 106 may be an indirect input device (eg, a mouse, keyboard, camera or camera array, etc.), or another type of non-tactile device, such as an audio input device.

[0067] The computing device(s) 102 may also include one or more input / output (I / O) interfaces 106 to allow the computing device 102 to communicate with other devices. The input / output (I / O) interfaces 106 may include one or more network interfaces to enable communication between the computing device 102 and other network devices, such as the other device(s) 102. The input / output (I / O) interfaces 106 may allow the device 102 to communicate with other devices, such as user input peripherals (e.g., keyboard, mouse, pen, game controller, voice input device, touch input device, gesture input device, etc.) and / or output peripherals (e.g., display, printer, audio speakers, tactile output, etc.).

[0068] 1.4 Exemplary Systems

[0069] Figure 2 2 is a schematic diagram of a system 200 that can be used to practice various multi-touch implementations as described herein. According to the example configuration, the system 200 includes a touch screen 202 and users 204 and 206 of the touch screen. The user 204 has a dominant hand 204a and a non-dominant hand 204b. The user 206 has a dominant hand 206a and a non-dominant hand 206b. (It should be noted that the user can wear the UWD on the non-dominant hand or the dominant hand). Such hand allocation is only an example, and the dominant hand and the non-dominant hand can be the left or right hand of the user. Although the fingers of the hand are shown to approach and / or touch the touch screen, the example includes more than one finger, the side or back of the hand or the thumb may approach or touch the touch screen, and the claimed subject matter is not limited thereto. Moreover, the right hand and left hand of the user can cross each other when they approach or touch the touch screen. In addition, the right hands and left hands of multiple users can cross each other when they approach or touch the touch screen.

[0070] System 200 may also include processor 210 associated with touch screen 202 and UWD 212. In various configurations, "system" may be considered to include any combination of things identified as being in system 200, which is a specific example, and the claimed subject matter is not limited thereto. For example, in some configurations, the system may be considered to be touch screen 202 and processor 210 (e.g., excluding users and UWDs). Again, the claimed subject matter is not limited thereto.

[0071] As used herein, unless otherwise specified, a "processor" may include one or more processors. Figure 1A processor 210 similar or identical to the processor 104 of the computing device 102 shown in FIG. 1 may be used to operate the touch screen 202. For example, the processor 210 may execute code to allow the touch screen 202 to display objects generated by any of a number of applications, which may also be executed by the processor 210. A memory 214, which may be locally accessible to the processor 210 (e.g., hardwired in the packaging of the touch screen 202 and / or the processor 210) or remotely accessible (e.g., in a wired or wireless computer network), may store such executable code or applications.

[0072] The UWD 212 may be communicatively coupled (e.g., wired or wirelessly) to the processor 210 (and thus to the touch screen 202). The UWD 212 may include a processor 216, a kinematic sensor 218, a memory 220, and a transmitter / transceiver 222. In some examples, the UWD 212 may also include a heartbeat monitor, a light sensor, a camera, a depth camera, and the like. The kinematic sensor 218, which may include an inertial sensor, a gravity sensor, a compass, an accelerometer, a barometric sensor, a force sensor or a strain gauge, a bending or flexibility sensor, a sensor that detects compression of a material, and the like, may generate kinematic data including the position, velocity, and / or acceleration of the UWD relative to an inertial coordinate system (e.g., with reference to the touch screen 202) and / or relative to a local coordinate system of the wearer itself. The UWD 212 may transmit the kinematic data to the processor 210 via the transmitter / transceiver 224, via the transmitter / transceiver 222. The memory 220 may store personal and / or identification data of an individual user. For example, such data may include user preferences for operating various applications (e.g., menu or display parameters), identification (e.g., ID number, name, username, etc.) to allow differentiation of the user from other users, and historical data of kinematic behavior (e.g., physical characteristics of a user's typical touch events). In some configurations, the UWD's GUID may be transmitted, while the rest of the data comes from the cloud or from the system 202, or a combination thereof. In some configurations, a user of the UWD 212 or system 200 must "opt in" or take other affirmative action before the UWD or system 202, or another system or cloud in communication with the UWD or system, uses or stores personal data.

[0073] In some configurations, the processor 210 may transmit information to the UWD 212 (e.g., Figure 2 ), such as handshake data or signals notifying the presence or various actions performed by processor 210 and / or UWD 212.

[0074] As described above, a touch event can be one or more hands or one or more fingers, etc., physically contacting the touch screen. Through any of the various techniques described herein, the system 200 can be able to identify specific users 204 and 206, and these users can have different attributes, such as different heights, different arm length ranges, etc. These attributes can be stored in the memory 214, 220 or the cloud (e.g., 103). For example, compared to user 204, user 206 can be taller than user 204 and can reach higher on the touch screen 202 toward the top 208 of the touch screen. User 206 can also have a wider reach, so that for a given standing or sitting position in front of the touch screen 202, hands 206a and 206b can be able to cover a wider portion of the touch screen 202 than user 204.

[0075] In some configurations, after determining which hand of the user is dominant and non-dominant, the system 200 can modify any of a number of features or objects displayed by the touch screen 202. These features or objects can include windows, menus, icons, brightness and / or contrast of particular portions of the touch screen, graphical objects, text, etc. For example, because hand 204a is the dominant hand of user 204, the area of ​​the touch screen 202 surrounding the location of user 204 or surrounding the touch event initiated by hand 204a can display objects appropriate for the motion of the dominant hand, which can be different from objects for the motion of the non-dominant hand.

[0076] In some configurations, after determining which hand of the user is left and right (e.g., independent of which hand is dominant and non-dominant), the system 200 can modify any of a plurality of features or objects displayed by the touch screen 202 based on such a determination. For example, a portion of the touch screen determined to be relatively close to the left hand (or within arm's reach, e.g., based on size information about the user, the user's handedness, etc.) can be displayed differently than a portion of the touch screen that is relatively close to the right hand.

[0077] In some configurations, using motion information provided by a UWD 212 worn by user 204 or 206 (or any possible additional users), the user can initiate a touch event involving two or more fingers, hands, or any combination thereof to virtually "grab" an object displayed by touch screen 202, and in real time, rotate and / or translate the displayed object as the user's hand or fingers (or wrist, etc.) rotates and / or translates accordingly. Such three-dimensional ("3D") manipulation of the displayed object can then be dragged and dropped to various portions of the touch screen's display, for example.

[0078] In some configurations, the 2D directional information provided by UWD 212 worn by user 204 or 206 (or any possible additional users) can be used in the example case where a finger / hand can act as if touching a (virtual) knob on a touch screen and turning the knob by rotating the finger / hand left or right.

[0079] In some configurations, directional information provided by a UWD 212 worn by user 204 or 206 (or any possible additional users) can be used to control the information displayed. If the user reaches a relatively high point (e.g., caused by a steeper tilt of the UWD), a particular menu can be displayed. If the user touches a relatively low point, a different menu can be displayed. In addition, rotation of the user's hand can generate additional menus. Depending on the location of the touch relative to the user's position, the UWD orientation can enable the display of multiple menus or different information.

[0080] In some configurations, orientation information provided by UWD 212 worn by user 204 and / or 206 (or any possible additional users) can be used to modify the touch event. For example, in a manner similar to retrieving new or deeper information through additional pressure, a user can orient a hand relative to a touch point (e.g., change the tilt and / or rotation of the hand, or the horizontal orientation of the hand) as a gesture to receive additional data related to the touched object. Pressure and orientation can also be combined in an instance.

[0081] In some configurations, using kinematic information provided by the UWD 212 worn by the user, the user can zoom in (enlarge) the displayed object. In some cases, the user can zoom in on the object while performing 3D manipulation on the object and / or other objects (e.g., air gestures). For example, the user 204 can initiate a touch event, including physical contact with the touch screen 202 or including hovering on the touch screen. The touch event can include the user performing a squeeze or stretch motion of their hands and / or fingers in space or on the surface of the touch screen. This motion can correspond to a negative or positive enlargement of an object displayed by the touch screen 202.

[0082] In some examples, the touch event may include a user's hand hovering over the touch screen. In this case, the processor may track the movement of the user's hovering hand and modify at least one object displayed by the touch screen based at least in part on the movement of the user's hovering hand. For example, the movement of the hovering hand may include a rotation of the hovering hand. Modifying the object(s) displayed by the touch screen may involve rotating the object(s) corresponding to the rotation of the hovering hand.

[0083] In some configurations, the system 200 need not include a camera, depth camera, or other image / video capture device to detect or measure the motion of one or more users by using kinematic data provided by the UWD 212 worn by the user. Such a configuration may also allow the system 200 to detect the user approaching and leaving the touch screen 202 (e.g., walking toward or away from the touch screen 202) using the UWD 212.

[0084] In some configurations, UWD 212 may provide personal information about each user (e.g., after each user "opts in" to choose), or may provide an identification of each user so that processor 210 may subsequently retrieve such personal information from memory 214.

[0085] In some configurations, the UWD 212 can allow personal information or user preferences to be transferred across multiple devices (e.g., devices other than the touch screen 202 and the processor 210). For example, if a user has a particular work-in-progress activity displayed on the touch screen 202, the user can transfer (e.g., leave the touch screen 202 and walk) to another computing device or display, and the work-in-progress activity can be similarly transferred to other computing devices or displays (e.g., the work-in-progress activity can be displayed by other devices). Icons, personal menus, display settings, etc. can be similarly transferred across multiple devices. The UWD's memory 220 can store data that enables such transfer. In another configuration, the UWD can be used as an identifier for a user of different devices so that different devices can retrieve (e.g., from another device on a network, a server, the Internet, or the cloud) icons, personal menus, settings, work-in-progress, etc.

[0086] All of the above configurations and capabilities can be used to implement various advantageous interactions between the user, the display, and the UWD. These are described in more detail in Section 1.5.

[0087] 1.4.1 Timing of touch events

[0088] Figure 3A timing diagram 300 (300a, 300b, 300c) includes an example touch event 302 performed by an arm / hand 304 wearing a UWD 306. In this example, the touch event 302 includes physical contact between a finger 308 and a display surface 310 of a touch screen 312. In other examples, instead of finger 308, any part of the arm / hand 304 (such as more than one finger or the side or back of the hand) can initiate such a touch event by physical contact with the display surface 310. Similarly, in some cases, there may be something in the hand touching the display surface 310 (e.g., a held stylus), which may result in different motion / acceleration profiles / spikes that can be identified using machine learning or the like (e.g., the spikes of a hand or (multiple) fingers without a stylus held in the hand may be different from the spikes of a hand or (multiple) fingers with a stylus held in the hand).

[0089] In detail, touch event 302 involves a time 314 before contact between finger 308 and surface 310, a time 316 at contact, and a time 318 after contact. Timing diagram 300a depicts the timing of touch event 302 represented by step 322. For example, a processor (e.g., processor 104 or 210) may operate an application programming interface (API) including touch screen 312 that detects the moment when finger 308 touches surface 310. Timing diagram 300b depicts the relative acceleration of the UWD during touch event 302 (hereinafter, the word "relative" is not necessarily explicit and may be implied), and is represented by curve 326. Timing diagram 300c depicts the timing of a threshold event represented by step 330, described below.

[0090] Prior to contact, the finger 308 (or any part of the arm / hand) approaches the surface 310 and strikes the surface 310 at a particular rate or velocity (e.g., rate is a scalar quantity that is independent of direction, while velocity is a vector quantity that includes magnitude and direction in each of three orthogonal directions). The UWD 306 generally follows the motion of the finger 308. For example, the rates of the UWD 306 and the finger 308 may be similar or identical, and the paths of the UWD 306 and the finger may be the same but offset by a fixed amount (e.g., the distance between the fingertip and the UWD). Upon contact with the surface 310, the finger 308 decelerates. In other words, the surface stops the motion of the finger. Hereinafter, unless otherwise noted, deceleration, i.e., negative acceleration, is referred to simply as "acceleration." The motion of the UWD 306 corresponds to the motion of the finger 308, and therefore the UWD 306 experiences similar acceleration. Over a very small time frame (between 50 ms and 100 ms in one example), the acceleration increases to a peak value and then decreases with a profile such as curve 326 (which is the acceleration profile of UWD 306), and can be described as a pulse. After the finger impacts the surface, a small rebound (e.g., a bounce) may occur, causing a reverse acceleration, which is shown by the portion of curve 326 below the time axis. Whether worn on an upper arm, wrist, or finger (or other part of the user), UWD 306 can experience an acceleration profile such as 326.

[0091] In some implementations, the UWD may be able to detect the presence of a touch surface through a proximity signal or possibly through capacitive coupling (for example). This detection can then wake up the sensing module to collect bursts of high-bandwidth sensor data. In other implementations, the UWD can initiate high-frequency sampling when the detection of a relatively large (e.g., "hard") acceleration interrupts the UWD's processor. For example, such an implementation can be used for power conservation in small battery-operated devices. In yet another implementation, the UWD can understand how the user touches the surface to predict and wake up the system at the appropriate time.

[0092] In some cases, finger 308 (or any part of an arm / hand) may approach surface 310 with a particular acceleration (e.g., scalar or vector). However, in the examples described herein, such "initial" acceleration may be negligible or ignored in a particular inertial reference frame, thereby assuming any initial acceleration to be zero.

[0093] Curve 326 can be described as a pulse having a particular shape based at least in part on the velocity and direction of finger 308 as it approaches and strikes surface 310. The pulse shape can also depend on which part of the user's arm / hand strikes the surface, whether the touch is made with the dominant or non-dominant hand, the position of the user wearing the UWD, the extension of the user's arm (e.g., elbow bend), the size of the user and other physical characteristics of the user, and habits or tendencies specific to a particular user, etc., as described below. For example, the velocity and direction of a finger on a dominant hand can be different from the velocity and direction of a finger on a non-dominant hand, and this difference can result in a different pulse shape.

[0094] In some examples, techniques for determining whether a touch event is generated by a user's dominant or non-dominant hand involve establishing a detection threshold (DT), which can be based, for example, on experiments and / or statistical data. DT is a specific value of acceleration that, when exceeded by a portion of an acceleration pulse of a touch event, indicates a certain condition different from a condition where the acceleration pulse does not exceed DT. For example, Figure 3 DT 332 is shown being surpassed by the peak portion of curve 326. This may indicate that UWD 306 is worn on the same arm / hand as finger 308 that performed touch event 302. If curve 326 has a relatively small peak amplitude, as shown below in Figure 5 , then DT 332 will not be overridden and it can be determined that the UWD is worn on a part of the user other than the same arm / hand as finger 308.

[0095] In some examples, the processor may determine that the curve 326 generates a touch event after the curve 326 rises above the DT 332. Specifically, when the curve 326 subsequently drops below the DT 332, the processor determines that the touch event 302 has occurred, as represented by step 330. In some implementations, the processor may consider a time limit measured from when the touch event 302 occurs, such as step 322. For example, if the acceleration pulse (e.g., represented by the curve 326) fails to exceed the DT 332 within such a time limit, the processor may determine that the acceleration pulse is initiated by the arm / hand opposite to the arm / hand wearing the UWD. In some examples, the failure of the acceleration pulse to exceed the DT 332 within such a time limit may indicate a situation where the acceleration pulse is negligible and is merely noise and is not caused by a touch event.

[0096] The shape of the acceleration pulse can indicate many things, as described below. For example, the acceleration pulse can include a specific shape or (multiple) features that allow the system 200 to determine that a particular user generates an acceleration pulse. In other words, a particular user can generate a unique and identifiable acceleration pulse caused by a specific movement habit or trend. The shape of the acceleration pulse can be characterized, among other things, by the slope, peak amplitude, full width at half maximum (FWHM) at various parts of the pulse, and a pulse width 334 at a DT such as 332. Of course, the claimed subject matter is not limited to such examples, and any of many other methods can be used to detect pulses, including methods involving machine learning.

[0097] Figure 4 A user's arms / hands 402 and 404 are shown interacting with example objects 406 and 408 displayed on a touch screen 410. In some examples, the UWD may be worn on the dominant arm / hand, and in other examples, the UWD may be worn on the non-dominant arm / hand. Figure 4 In the example shown, arm / hand 402 is considered the dominant arm / hand and is wearing UWD 412. Arm / hand 404 is considered the non-dominant arm / hand and is not wearing a UWD.

[0098] Objects 406 and 408 may include any of a number of displayable things, such as windows, menus, text, drawings, icons, images, etc. For example, if object 406 is a menu, object 414 may be one of a plurality of menu items that may be selected by a user touching object 414. In some examples, object 408 may include text that includes a plurality of words 416 that may be selected by a user touching any of the words.

[0099] As described above, each hand of the user can have a different role when interacting with an object or device. For example, non-dominant hand 404 can be dedicated to positioning object 408 for dominant hand 402. In some configurations, the system can automatically distinguish between the user's dominant hand and non-dominant hand by receiving a wireless signal transmitted to the system from UWD 412 that represents the movement of the UWD. The movement of the UWD can correspond to the movement of hands 402 and 404 (although the movement correspondence between hand 402 and UWD 412 can be closer than the movement correspondence between hand 404 and UWD 412). It should be noted that the right hand or the left hand can be the user's dominant hand.

[0100] Figure 5Two different users' arms / hands 502 and 504 are shown interacting with example objects 506 and 508 displayed on a touch screen 510. In some examples, UWD 512 can be worn on the (dominant or non-dominant) arm / hand of a first user, and UWD 514 can be worn on the (dominant or non-dominant) arm / hand of a second user. Objects 506 and 508 can include any of a number of displayable things, such as windows, menus, text, drawings, icons, images, etc.

[0101] In some implementations, the system can determine which of the two users is performing a touch event using the touch screen. For example, UWD 512 and 514 can provide the system with identification data of their corresponding wearers. The system can associate the touch event (e.g., its timing) with the movement of the UWD and the identification data they provide. For example, UWD 512 can provide such identification data to the system periodically, from time to time, in response to a request from the system, or at the time of a touch event. If the system senses a touch event (e.g., an electronic device via a touch screen), the system can associate the touch event with the movement of UWD 512 and the movement of UWD 514 to determine which user initiated the touch event. The correlation between the trigger event and the movement of the UWD worn on the arm / hand that initiated the touch event will be stronger than the correlation between the touch event and the movement of the UWD worn on the arm / hand that did not initiate the touch event. Therefore, the system can determine which user initiated the touch event. Although this example involves two users, this method can be performed, for example, by the system for any number of users interacting with the touch screen.

[0102] 1.5 Multi-touch detection implementation

[0103] The exemplary operating environment 100 and the exemplary system 200 described above can be used to implement various multi-touch detection implementations as described herein. The following multi-touch detection implementation provides various tools for improving the usability of display devices (e.g., large touch screen devices). These tools include user-associated personalized mobile menus, combined click and radial marking menus, menus for automating and improving the drawing or manipulation of content on the display, new menus and methods for selecting objects and text on the display, and new interactions with UWD and touch screen displays using gestures, etc. In addition, targeted tactile feedback to the UWD of a specific user of the display is enabled. Tactile feedback in the form of vibration can be provided to the UWD of a specific user, for example, in one instance, when he / she aligns an object to a grid.

[0104] Some multi-touch detection implementations described herein utilize UWD as described above in conjunction with the display to implement a new and advantageous interface with a large or small touch screen display. In many multi-touch detection implementations, a UWD as described herein is used (and / or via other methods such as by determining the user's fingerprint and / or by using a camera to track the user) to determine the user's dominant and non-dominant hands. This allows determining when the user's non-dominant hand (or dominant hand) touches the touch screen display, as discussed in the above paragraph. In some multi-touch detection implementations, the user's non-dominant hand wears a UWD (e.g., a band) and uses an accelerometer and / or one or more other sensors on the UWD to determine when the user's non-dominant hand contacts or touches the touch screen display. In other multi-touch detection implementations, the user's dominant hand wears a UWD (e.g., a band) and uses an accelerometer and / or one or more other sensors on the UWD to determine when the user's dominant hand contacts or touches the touch screen display. Alternatively, in some implementations, a sensor on a device being held or touched, or a sensor on a UWD, or a combination of both can be used to determine which hand is dominant and non-dominant and which hand is touching or otherwise acting on the touch screen display.

[0105] 1.5.1 Mobile Personalized Menu

[0106] Many multi-touch detection implementations described herein provide menus that move as a user's finger touches the display (in some implementations, as long as there is the same touch event). This is advantageous, for example, because the user can move freely about a large display without having to reach the menu to add or manipulate objects or data. In some implementations, when it is determined that a finger or more than one finger of a user's (e.g., non-dominant) hand touches the touch screen, a menu is displayed on the display at the position of the (multiple) finger on the display. In some implementations, as long as the finger touches the display, the menu follows the finger of the user's (non-dominant) hand throughout the display. In some multi-touch detection implementations, if the user lifts this finger of the (non-dominant) hand from the display (e.g., there is no longer a touch event), the menu disappears from the display. In some implementations, the user's non-dominant hand can be used to activate various options on the menu (e.g., using a finger on the non-dominant hand), or the user can use a pen in their dominant hand or add objects or strokes with a finger of their dominant hand. It should be noted that the roles of the dominant hand and the non-dominant hand in menu display and manipulation can be reversed.

[0107] In some multi-touch detection implementations described herein, when a user touches a display, a personalized menu can be displayed to each user. For example, as described above, a UWD can be used to determine the identity of a user, and the determined identity can be used to associate a personalized menu and personal data with a specific user. Therefore, two or more different users can manipulate data (e.g., files, images, search results) and objects on a large display at the same time, each user using their own personalized data and personalized menus (e.g., menus most frequently used by users, menus configured to meet user preferences, such as the preferred shape of a curve board, preferred dictionaries for handwriting recognition, the most recent color and / or pen attributes automatically selected when the pen unique ID is associated with a specific user, etc.). Moreover, in some implementations, personalized menus and preferences can seamlessly follow users across different devices (i.e., by storing / retrieving the information via a computing cloud). In one instance, this can be achieved by having the UWD transmit credentials and / or user information to a computing device / processor connected to a display (or remotely connected to a display), and automatically and securely using the information to retrieve data from a computing cloud (i.e., using a Web service, etc.). The data can then be displayed near the user. Personalized menus may also be mobile, such that they move with touch events as described above, or may be non-mobile, such that they do not move.

[0108] 1.5.2 Combined single-click radial marking menu

[0109] Some multi-touch detection implementations described herein use combined click radial marking menus. These combined click radial marking menus can be used as ordinary radial marking menus, but combined click radial marking menus can also be allowed to be positioned above an object that a user wishes to act on by touching the touch screen or then displaying a radial marking menu icon (e.g., with a finger of the user's non-dominant hand). The user can then expand the radial marking menu and select a menu option from the radial marking menu by touching the menu icon in an outward motion from the center of the radial marking menu on the desired menu selection (e.g., with their dominant hand or a pen). Thus, the user can modify the object below the combined click radial marking menu with the selected menu option. For example, the user can modify the stroke color or stroke width of the line below the menu. In some implementations, the combined click radial marking menu can be expanded from an unexpanded state (represented by an icon) by applying a certain degree of pressure to the touch display. In other implementations, the user can twist their non-dominant hand (which can be sensed by the UWD) to expand the menu. In some implementations, a combined click on a radial marking menu modifies the color of the stroke(s) immediately below the menu, and the radial marking menu selection modifies the persistent state of the pen (e.g., the color of that particular pen, which may be determined in a variety of ways). Of course, many other stroke attributes may also be modified, such as stroke thickness, stroke style (e.g., translucent highlighter, marker, dashed line, air spray, etc.). For example, if the state of the pen is green (e.g., the most recent radial menu selection was green), and the user clicks to modify the existing stroke to red, the state of the stylus remains green (until the next different radial menu selection). In some implementations, the click action or menu is visually identified by a translucent menu on the display.

[0110] Figure 6 Depicted is an exemplary combined tap and radial marking menu 600 for modifying stroke characteristics (e.g., width) of a stroke 602 drawn on a touch screen display 604. In some implementations, the menu 600 appears on the display 604 when a user touches the display 600 with their non-dominant hand as determined by the UWD 608 (which can be referred to as a non-dominant touch). The user can then modify the stroke 402 by selecting a menu item with the user's dominant hand (which can be referred to as a dominant touch). Figure 7 Depicted in Figure 6 702 (e.g., the stroke is thicker) of a stroke drawn on the touch screen display depicted in FIG. 703. It should be noted that this example assumes that the user's non-dominant hand is the left hand, but a symmetrical menu may appear if the non-dominant hand is the right hand (in one implementation, the UWD may inform the display / processor of the computing device of the dominant hand).

[0111] 1.5.3 Content / object selection and manipulation using the X-ray menu

[0112] Some multi-touch detection implementations provide for selecting content or objects on the screen and actions on them using an x-ray menu. Typically, the x-ray menu combines the selection of a range (e.g., which (which) strokes or other objects recognize or otherwise act on) with the expression of the command itself. Compared to other methods of selecting items on the display (e.g., switching to lasso mode and then drawing the strokes / objects to be selected with long strokes), the x-ray menu requires less user interaction and is therefore computationally efficient. In some implementations, the x-ray menu allows the desired content to be selected by selecting objects or content in an area defined by multiple fingers on the user's (e.g., non-dominant) hand (e.g., two fingers or fingers and thumb of the user's non-dominant hand). One or more menus that allow manipulation of the selected object or content on the display associated with the selected content / object may appear at the edge of the selected content or object (or at other locations, e.g., next to the selected content, close to the non-dominant finger, at the top or bottom of the selection, etc.). In some implementations, the selected area gradually moves away from the intersection of the finger / fingers of the user making the selection. Additionally, in some implementations, the distance that the selected area extends from the finger / finger can be increased by increasing the pressure applied to the touch screen by the finger / finger touching the touch screen. In some implementations, the selected area extends when the user performs certain gestures with the finger or pen (e.g., the dominant hand moves outward on the x-ray to extend it, and inward to retract it). Additionally, the selection of content or objects can be redirected by repositioning the finger / finger touching the screen.

[0113] Figure 8 Depicted is a selection of a display area including content (e.g., the word "slope") on a touch screen 800, which includes unexpanded menus 804, 806 associated with the selected area. In other implementations, the location of these menus may be different. In some implementations, the selected area and associated display are initiated by touching the display 800 with a finger of the user's non-dominant hand. This example shows the selection of the handwritten text "slope" and the display of an associated menu for applying text recognition to the handwritten text.

[0114] Fig. 9 Depicted Figure 8 9 and 9. The combined click and expansion of the radial text recognition menu 902 shown in FIG. In some implementations, the menu can be expanded by the user pressing the center of the menu with a finger or stylus / pen and holding the center of the menu down. Fig.10 Describes the use Figure 8 and 9The combined click radial marking menu shown in activating the application of text recognition converts the handwritten text "slope" to the type writing "slope" 1002. In some implementations, there is an "unrecognized" menu that allows the recognition to be restored. It should be noted in this example that if the text is not horizontally oriented, the orientation of the user's two fingers / fingers that define the orientation of the x-ray can also be used to correctly orient the selection of the text to be recognized. In one instance, the recognized text can maintain the same orientation (and size, and other characteristics, such as bold, color, italics, etc. in the case of thick strokes) as the handwritten text.

[0115] There are also various other ways in which a user can select content in the various multi-touch input detection implementations described herein. In some multi-touch detection implementations, the area selected for editing or modifying an object is not defined by the area between two fingers of a user's (e.g., non-dominant) hand, but rather by a fixed distance from the touch of the fingers of the (e.g., non-dominant) hand. For example, a radius area of ​​a given distance around the location of the touch of the user's fingers can be used to select objects and content displayed on the display. An applicable menu for manipulating the selected content can also be displayed (e.g., at the boundary of the radius of the circle defining the selected content).

[0116] 1.5.4 Menu outline as tool

[0117] In some multi-touch detection implementations, the menu outline is an implicit part of the tool palette / menu itself, which allows the user to use the pen to draw a curve or line that conforms to the shape of the outline of the menu / tool ​​palette by bringing the pen stroke reasonably close (e.g., within a specified distance) to the menu / tool ​​palette itself. The shape of the line is not provided by a separate mode or tool that can be selected from the palette itself; instead, it is just there by default and is actually a spatial multiplexing function that the user accesses simply by bringing the pen stroke reasonably close to the appropriate boundary of the tool palette. This is different from the previous curve board, curve guide, and ruler tools, in which the tool always existed first as some kind of control that had to be selected from the palette (or toolbar, or wherever the function resides in the graphical user interface). This design sidesteps this, and the menu / tool ​​palette itself implements the behavior based solely on the outer outline of the entire menu / tool ​​palette.

[0118] Fig.11 A menu 1100 is depicted having an outline 1102 for use as a tool for drawing a curve board or other shape. The outline 1102 of the menu 1100 can be used as a tool without having to call up a specific drawing tool menu or control. Fig.11A user is depicted drawing a curve 1104 that follows the outline 1102 of the menu 1100 by drawing a stroke only near the outline of the menu. In one implementation, the menu is activated by a finger of the user's (e.g., non-dominant) hand 1106 touching the display. In some implementations, the orientation of the entire menu 1100 can be rotated by the user, but in some cases, the radial menu (within the menu 1100) retains its orientation (e.g., the north menu selection remains pointing upward). In other cases, the radial menu rotates with the menu 1100 (so that when the menu 1100 rotates, the north mark on the radial menu is in the northwest (NW) direction).

[0119] Fig.12 Depicted Fig.11 1004. The user modifies curve 1004 by modifying the characteristics of the curve using touch or multi-touch (i.e., with the user's dominant hand). In some implementations, the user can change the shape of the curve to be drawn by changing the outline of the menu itself. In this case, any strokes drawn within a specified distance of the modified menu outline will conform to the new modified menu shape. These modifications can be personalized for the user who made them, and can follow the user when the user is using different devices. In addition, in one instance, the modification history of the curve board tool can be stored, or different custom curve board tools can be set by the user to different preferences for shortcut menus, etc.

[0120] In some multi-touch detection implementations, the outline of the displayed mobile menu is used as other types of tools. For example, in some implementations, the outline of the menu is used as a ruler or ruler. In addition, in some implementations, the edge of the menu can be used as an alignment tool, for example, as a boundary for dragging an object. Alternatively or additionally, the menu outline can be used as a frisket (mask). The menu outline can also be used to enforce a specific type of constraint or symmetry, such as a mirror reflection of the stroke of the current pen. In some multi-touch detection implementations, the menu outline can be used to align the ruler to 90 degrees or 45 degrees increments by default. In addition, modified gestures can be used to allow users to break constraints and place the ruler at any angle. In some multi-touch detection implementations, the menu outline can be used to provide a compass or protractor edge. In another example, the edge can apply a line style, etc. (i.e., a dotted line, a line with a grid mark, etc.). These implementations are advantageous because they make user input simpler and computationally more efficient, and the user never feels like they have to "select" or use a tool mode (they still just put down strokes with the pen, it feels like a completely modeless approach), the added functionality is always there and can be accessed in a seamless manner when needed.

[0121] 1.5.5 Gestures combined with UWD for menu control

[0122] In some multi-touch detection implementations, gestures are used to control the displayed menu. For example, in some multi-touch detection implementations discussed previously, when a user touches the touch display with a finger of his or her non-dominant hand, a menu pops up at the location where the finger touches. In some implementations, when the user no longer touches the display, the menu usually disappears. However, in some multi-touch detection implementations, the user can perform gestures in a predefined manner to manipulate the menu on the display. For example, in some multi-touch detection implementations, the user can lock the displayed menu in a position on the display by twisting the finger, hand or arm of the non-dominant finger that is touching the display and popping up the menu on the display. In some implementations, visual feedback (e.g., a change in color, a change in transparency) is provided to indicate that the menu has been locked. The user's finger distortion can be determined by an accelerometer and / or one or more other sensors on the UWD that the user is wearing. In this case, the displayed menu is locked in position on the display and remains displayed until the user chooses to move the menu or close it, such as by touching the menu again with a finger of the non-dominant hand and then removing their fingers (e.g., without distortion) from the display. Other gestures can be used to execute other commands on the menu. For example, a user may raise a hand in a gesture to cause a menu to disappear, expand a menu, and make a menu selection.

[0123] 1.5.6 User-specific tactile feedback combined with UWD

[0124] In some multi-touch detection implementations, the user can not only receive data from the UWD, but also commands can be sent to the UWD to perform mechanical actions by the UWD. In some implementations, the UWD includes a sound and / or vibration generating device, which can prompt the user (and / or other users) whether the user's arm / hand has taken a specific action on a specific object on the display. For example, in some implementations, tactile feedback can be sent to the user for the action taken by the user for the display. For example, when the user changes the menu, selects a specific menu, aligns one or more strokes to the menu outline, aligns an object (or a group of objects) to a grid, or takes other actions on an object or menu displayed on the display, the user can receive tactile feedback on his UWD. In another implementation, the first user can align objects and receive tactile feedback, but other users who touch the display (and / or are close to the display) will receive different feedback (i.e., simultaneous notifications) or no tactile feedback at all. In another example, the user who touches the display will receive different feedback from other users who are near the display (e.g., within a specified distance from the display) but do not touch the display. In yet another example, the actual distance from the user to the display affects the type and / or intensity and / or frequency, etc. of the haptic feedback (ie, if the user is far away from the display, very little haptic feedback is provided).

[0125] Since the identity of the user is known because it is associated with his UWD, the tactile feedback can be personalized based solely on the actions taken by that particular user. It should be noted that this tactile feedback or auditory prompt sent to the UWD can occur simultaneously with a touch event on the touch screen, but can also occur after or before such a touch event, or may not be associated with a touch event at all. In some implementations, the user performing the operation will not receive tactile feedback, but other users can receive tactile feedback. In some cases, tactile feedback is sent during operations using a hand that is not wearing a UWD. In some implementations, tactile feedback can also be associated with hover events (e.g., hovering with a stylus over a grid).

[0126] 1.5.7 Portability to smaller screens and distributed processing using smaller screens

[0127] In some multi-touch detection implementations, the menus and tools described herein displayed on a large touch display can be partially or completely ported to a smaller display or another device, such as a mobile phone or a handheld tablet device (or vice versa). For example, in these exemplary implementations, menus and tools displayed on a large screen device can be ported to a handheld device. Many of the same interactions discussed herein can be supported in such a context, which allows the user (for example) to utilize some of the capabilities of the multi-touch detection implementation described herein, even when the user is slightly beyond the arm's length of the display. In some implementations, the entire menu or part thereof can be collated to the display of a UWD and / or a mobile phone and / or some other wearable and / or non-wearable mobile device (for example, a tablet computer, a watch, or possibly even to the pen itself). In one implementation, the system knows which user touches the display due to the UWD, so this information allows a connection to a device associated with the user (such as a tablet computer, a mobile phone, or a pen), and a menu or a portion of a menu is collated to a device associated with the user. For example, if the device is a pen, user preferences (i.e., the style, color, size of the stroke, etc.) can be automatically set. In some implementations, a touch-sensitive device, such as a touchpad on a computing device, can be used by the user's non-dominant hand, and a mouse can be used by the user's dominant hand to control a non-touch display or a touch display, such as a large display in a conference room. In addition, the correlation between the movement of the mouse (or touch events on the touchpad) and the movement of the UWD associated with the user can be used to seamlessly authenticate the user.

[0128] Fig.13 A UWD 1302 is shown that is worn by a user on an arm / hand 1304, which interacts with a handheld touch screen 1306, which can be used to port some of the tools and menus described herein to another device for display. For example, a user's personalized menu can be ported to other display devices of the same size, or can be ported between large-screen and small-screen displays. Typically, menus can be distributed across all of a user's devices, regardless of their shape, sensing capabilities, and size. Moreover, in one implementation (e.g., on a desktop display), the system can recognize when a user places a mobile device (e.g., a mobile phone) down on the display (i.e., by determining a specific spike in an accelerometer and / or gyroscope on the device and the footprint of the mobile device on the display), and automatically create a hybrid experience, such as causing a curved board customized for the user to appear around the mobile device on the desktop display, and causing other menus to be displayed on the mobile device itself.

[0129] 1.5.8 Example Process

[0130] Fig.14is a flow chart of an exemplary computer-implemented process 1400 that can be executed by a processor. For example, the process 1400 can be executed by Figure 1 The computing device 102 shown in FIG. 1 is executed.

[0131] At block 1402, a processor receives a touch event from a user on a touch display in communication with a computing device.

[0132] At block 1404, as long as the touch event is active, a tool is displayed on the touch display at the location of the touch event, wherein the tool provides the ability to modify an object displayed in the specified vicinity of the touch event on the touch display. The tool may be fixed to the display even when the touch event ends in response to a user gesture.

[0133] Fig.14 The operational flow shown in is shown as a collection of boxes and / or arrows representing a sequence of operations that can be implemented with hardware, software, firmware, or a combination thereof. The order of the description boxes is not intended to be interpreted as a limitation, and any number of the described operations can be combined in any order to implement one or more methods or alternative methods. In addition, without departing from the spirit and scope of the subject matter described herein, separate operations can be omitted from the operational flow. In the context of software, a box represents a computer-readable instruction that configures a processor to perform the operation when executed by one or more processors. In the context of hardware, a box can represent one or more circuits (e.g., FPGA, application-specific integrated circuit ASIC, etc.) configured to perform the operation.

[0134] Fig.14 Any routine descriptions, elements or blocks in the operational flows shown in the drawings may represent modules, segments or portions of code including one or more executable instructions for implementing specific logical functions or elements in the routines.

[0135] 2.0 Sample Clauses

[0136] The following paragraphs describe various multi-touch detection implementations.

[0137] A. A system comprising: a touch screen; one or more processors; a transceiver for receiving signals from a device associated with a user and sending signals to a device associated with the user; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving data representing acceleration of a sensor residing in a device associated with the user during a touch event of the touch screen; determining, based at least in part on the received data, whether the touch event is associated with a conventional input or whether the touch event is associated with a non-conventional input; initiating a first command if the touch event is associated with a non-conventional input; and initiating a second command if the touch event is associated with a conventional input.

[0138] B. A system according to paragraph A, wherein the stored instructions, when executed by the one or more processors, cause the one or more processors to perform operations including: displaying at least one menu at the location of the touch event on the touch screen in response to the first command.

[0139] C. The system of paragraph B further includes using the second command to modify at least one object displayed by the touch screen using the at least one displayed menu.

[0140] D. A system according to paragraph B, wherein the menu moves as the touch event moves across the display as long as the touch event remains active because the user or other object is touching the touch screen.

[0141] E. A system according to paragraph B, wherein the menu is personalized for a user associated with the first touch event.

[0142] F. A system according to paragraph B, wherein the at least one menu also includes at least one radial marking menu for modifying objects located below the menu on the touch screen.

[0143] G. A system according to paragraph B, wherein the touch event is determined to be caused by a specified number of fingers of the user's hand touching the touch screen, and wherein an object in the area between the specified fingers is selected and modifiable through a menu associated with the selected area on the touch screen.

[0144] H. A system according to paragraph B, wherein the touch event is determined to be caused by the user's hand touching the touch screen, and wherein one or more objects in the area surrounding the touch event are selected and modifiable through a menu associated with the selected area on the touch screen.

[0145] I. A system according to paragraph B, wherein the outline of the border of the menu is used as a tool to act on objects displayed on the touch screen.

[0146] J. A system according to paragraph I, wherein the outline of the border of the menu is modifiable by the user.

[0147] K. A system according to paragraph A, wherein one or more commands to a device associated with the user are wirelessly transmitted to the device associated with the user via the transceiver.

[0148] L. A system according to paragraph K, wherein the one or more commands are tactile input.

[0149] M. A system as described in paragraph K, wherein the one or more commands are targeted to a device associated with a particular user.

[0150] N. A system according to paragraph A, wherein the dominant input is associated with a dominant hand of a user of the touch screen and the non-dominant input is associated with a non-dominant hand of the user.

[0151] O. A system according to paragraph A, wherein the stored instructions, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving data representing an identification of a user of the associated device from a device associated with the user; and modifying a menu displayed on the touch screen based at least in part on the identification of the user.

[0152] P. A system as described in paragraph A, wherein the device associated with the user is a user-wearable device.

[0153] Q. A system comprising: a first touch screen; a receiver for receiving a signal from a user wearable device; a processor communicatively coupled to the receiver to: receive data representing acceleration or other motion data of the user wearable device during a touch event performed by the user on the first touch screen; determine which hand of the user generated the touch event based at least in part on the received data; and display a tool associated with the user at a location on the touch screen corresponding to the location of the touch event on the first touch screen based at least in part on the determination of which hand of the user generated the touch event; and transplant the displayed tool associated with the user at the location in the first touch screen to a similar relative location on a second touch screen.

[0154] R. A system according to paragraph Q, wherein the processor is also communicatively coupled to a memory to: store at least a portion of the acceleration or other motion data and associate the stored acceleration or other motion data with the user; store additional acceleration or other motion data of a second user of the touch screen and associate the stored additional acceleration or other motion data with the second user; and access the stored acceleration or other motion data and the stored additional acceleration or other motion data to determine whether the touch event was generated by the user or the second user.

[0155] S. A system according to paragraph R, wherein the processor is configured to modify at least one object displayed by the touch screen based at least in part on determining whether the user or the second user generated the touch event.

[0156] T. A computer-implemented process method comprising using a computing device for: receiving a touch event from a user on a touch display in communication with the computing device; and displaying a tool on the touch display at an approximate location of the touch event as long as the touch event is active, wherein the tool provides the ability to modify an object displayed in a specified vicinity of the touch event on the touch display.

[0157] U. A computer-implemented process as described in paragraph T, wherein the tool can be fixed on the touch display even when the touch event ends through the use of a user gesture.

[0158] V. A computer-implemented process as described in paragraph T, wherein the tool moves with the location of the touch event as long as the touch event is active.

[0159] Although the techniques have been described in language specific to structural features and / or methodological acts, it should be understood that the appended claims are not necessarily limited to the described features or acts. Rather, the features and acts are described as example implementations of the techniques.

[0160] Unless otherwise specified, all of the above methods and processes may be implemented in whole or in part by software code modules executed by one or more general-purpose computers or processors. The code modules may be stored in any type of computer-readable storage medium or other computer storage device. Alternatively, some or all of the methods may be implemented in whole or in part by dedicated computer hardware such as FPGA, ASIC, etc.

[0161] Unless otherwise specifically stated, conditional language such as "can," "could," "might," or "may" is used to indicate that some examples include and other examples do not include the described features, elements, and / or steps. Therefore, unless otherwise stated, such conditional language is not intended to imply that one or more examples require these features, elements, and / or steps in any way, nor is it intended to imply that one or more examples must include logic for deciding whether these features, elements, and / or steps are included or to be performed in any particular example, with or without user input or prompting.

[0162] Unless expressly stated otherwise, linking language such as the phrase "at least one of X, Y, or Z" should be understood to mean that an item, term, etc. can be X, or Y, or Z, or a combination thereof. In addition, in some cases, "A and B" can also mean "A and / or B".

[0163] Many variations and modifications may be made to the above examples, and the elements thereof should be understood as other acceptable examples. For example, various types of menus and tools may be implemented using the multi-touch detection implementation described herein. For example, Fig.15 Depicted is an exemplary expansion of a combined point-and-click and radial marking menu. Fig.16 Depicted are exemplary menus that may be accessed via a combined click and expansion of a radial marking menu employed in some multi-touch detection implementations described herein. Fig.17 A combined click and radial menu of photos from which a particular user can be selected is depicted. In some implementations, these photos can be retrieved from a computing cloud (e.g., the cloud). All such modifications and variations are intended to be included within the scope of this disclosure.

Claims

1. A system comprising: First touch screen; A receiver, for receiving a signal from a user's wearable device; as well as a processor, communicatively coupled to the receiver to: receiving data representing acceleration or other motion data of a wearable device of the user during a touch event performed by the user on the first touch screen; determining which hand of the user generated the touch event based at least in part on the received data; as well as displaying at least one tool associated with the user at a location on the first touch screen corresponding to a location of the touch event on the first touch screen based at least in part on the determination of which hand of the user generated the touch event; as well as The at least one tool displayed associated with the user at the location on the first touch screen is transplanted to a similar relative location on the second touch screen.

2. The system of claim 1 , wherein the processor is further communicatively coupled to the memory to: storing at least a portion of the acceleration or other motion data and associating the stored acceleration or other motion data with the user; storing additional acceleration or other motion data of a second user of the touch screen and associating the stored additional acceleration with the second user; as well as The stored acceleration or other motion data and the stored additional acceleration or other motion data are accessed to determine whether the touch event was generated by the user or the second user. 3 . The system of claim 2 , wherein the processor is configured to modify at least one object displayed by the touch screen based at least in part on determining whether the touch event was generated by the user or the second user.

4. A method occurring on a computing system, the computing system comprising: First touch screen; A receiver, for receiving a signal from a user's wearable device; and a processor communicatively coupled to the receiver, the method comprising: receiving data representing acceleration or other motion data of a wearable device of the user during a touch event performed by the user on the first touch screen; determining which hand of the user generated the touch event based at least in part on the received data; and Based at least in part on the determination of which hand of the user generated the touch event, displaying at least one tool associated with the user at a location on the first touch screen corresponding to a location of the touch event on the first touch screen; and The at least one tool displayed associated with the user at the location on the first touch screen is transplanted to a similar relative location on the second touch screen.

5. The method of claim 4, wherein the processor is further communicatively coupled to a memory, the method further comprising: storing at least a portion of the acceleration or other motion data and associating the stored acceleration or other motion data with the user; storing additional acceleration or other motion data of a second user of the touch screen and associating the stored additional acceleration with the second user; as well as The stored acceleration or other motion data and the stored additional acceleration or other motion data are accessed to determine whether the touch event was generated by the user or the second user.

6. The method according to claim 5, further comprising: At least one object displayed by the touch screen is modified based at least in part on determining whether the touch event was generated by the user or the second user.

7. A computer-readable storage device having stored thereon instructions that, when executed by a computing system comprising a first touch screen, a receiver for receiving a signal from a user wearable device, and a processor communicatively coupled to the receiver, cause the computing system to perform operations comprising: receiving data representing acceleration or other motion data of a wearable device of the user during a touch event performed by the user on the first touch screen; determining which hand of the user generated the touch event based at least in part on the received data; as well as displaying at least one tool associated with the user at a location on the first touch screen corresponding to a location of the touch event on the first touch screen based at least in part on the determination of which hand of the user generated the touch event; as well as The at least one tool displayed associated with the user at the location on the first touch screen is transplanted to a similar relative location on the second touch screen.

8. The computer readable storage device of claim 7, wherein the processor is further communicatively coupled to a memory, the operations further comprising: storing at least a portion of the acceleration or other motion data and associating the stored acceleration or other motion data with the user; storing additional acceleration or other motion data of a second user of the touch screen and associating the stored additional acceleration with the second user; as well as The stored acceleration or other motion data and the stored additional acceleration or other motion data are accessed to determine whether the touch event was generated by the user or the second user.

9. The computer readable storage device of claim 8, wherein the operations further comprise: At least one object displayed by the touch screen is modified based at least in part on determining whether the touch event was generated by the user or the second user.

Citation Information

Patent Citations

  • Determining User Handedness and Orientation Using a Touchscreen Device

    US20150261373A1