Automatically correct touchscreen errors

Abstract

Techniques are provided for correcting touchscreen errors. A first position input is received from a user touching a global touchscreen of a global device, where the first position input is a position relative to a display surface of the global touchscreen. A motion of the global device is determined to be different from a motion of a local device. Motion information of the local device is retrieved. The first position input is corrected to a second position input based on the motion information. An action is performed in response to the second position input.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present invention relate to automatically correcting touchscreen errors, and more particularly, to automatically correcting touchscreen errors on a device via an external or non-worn source of information. [Background technology]

[0002] There are strategies and systems for detecting and correcting typing errors when using a device's touchscreen. Some strategies are based on human behavior and general interaction with the touchscreen and its keyboard in particular, while other strategies are based on motion sensors (e.g., accelerometers).

[0003] These strategies are useful in detecting situational failures by leveraging motion sensors within the device exposing the touchscreen. These strategies are based on the assumption that the relative motion of the finger and the touchscreen is identical, allowing errors due to vibrations caused by walking or lack of attention or both to be corrected by the device's internal motion sensors.

[0004] For example, a person is walking while holding a smartphone with a touch screen. The motion sensor inside the smartphone measures the acceleration of the person by assuming that the smartphone and the person are one unique rigid body. Thus, the inertia that causes the relative delta motion of the finger on the touch screen is proportional to the acceleration measured by the smartphone's motion sensor. Summary of the Invention

[0005] According to one embodiment, a computer-implemented method for automatically correcting touchscreen errors is provided. The computer-implemented method includes operations: a first position input is received from a user touching a global touchscreen of a global device, where the first position input is a position relative to a display surface of the global touchscreen; a motion of the global device is determined to be different from a motion of the local device; motion information of the local device is retrieved; the first position input is corrected to a second position input based on the motion information; and an operation is performed in response to the second position input.

[0006] According to another embodiment, a computer program product is provided for automatically correcting touchscreen errors. The computer program product includes a computer readable storage medium having program code embodied therein, the program code being executable by at least one processor to perform an operation. A first position input is received from a user touching a global touchscreen of a global device, where the first position input is a position relative to a display surface of the global touchscreen. It is determined that a motion of the global device is different from a motion of the local device. Motion information of the local device is retrieved. The first position input is corrected to a second position input based on the motion information. In response to the second position input, the operation is performed.

[0007] According to yet another embodiment, a computer system for automatically correcting touchscreen errors is provided. The computer system includes one or more processors, one or more computer readable memories, one or more computer readable tangible storage devices, and program instructions stored in at least one of the one or more computer readable tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more computer memories to perform an operation. A first position input is received from a user touching a global touchscreen of a global device, where the first position input is a position relative to a display surface of the global touchscreen. It is determined that a movement of the global device is different from a movement of the local device. Motion information of the local device is retrieved. The first position input is corrected to a second position input based on the motion information. In response to the second position input, an operation is performed.

[0008] Reference is now made to the drawings in which like reference numbers represent corresponding parts throughout. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates, in block diagram form, a computing environment in accordance with one embodiment. [Diagram 2] FIG. 2 illustrates a device in a car according to an embodiment. [Diagram 3] FIG. 3 is a diagram illustrating the movement of devices within a car according to one embodiment. [Figure 4] FIG. 4 illustrates a global device and a local device according to one embodiment. [Diagram 5] FIG. 5 illustrates in a flow chart the operations for device registration according to one embodiment. [Figure 6A] FIG. 6A illustrates registration, monitoring and correction according to one embodiment. [Figure 6B] FIG. 6B illustrates registration, monitoring and correction according to one embodiment. [Figure 6C] FIG. 6C illustrates registration, monitoring and correction according to one embodiment. [Figure 7] FIG. 7 illustrates a computing environment according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosed embodiments. The terms used in this specification are selected to explain the principles of the embodiments, practical applications, and technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0011] Embodiments automatically correct for typing errors on a touchscreen of a device that does not have a motion sensor (e.g., an accelerometer) or where the device's internal motion sensors are not directly causally related to the user's movements. For example, embodiments automatically correct for touchscreen errors in the following scenarios: The touchscreen and the user using the touchscreen are not a single rigid body, so the relative movements of the user's finger and the touchscreen are not the same; The device does not have a motion sensor installed; More than one user may use the same touchscreen simultaneously.

[0012] FIG. 1 illustrates in a block diagram a computing environment according to an embodiment. A vehicle 100 includes a global device 110 and one or more local devices 150a,...,150n. The vehicle 100 may be a car, an electric bike, an airplane, a truck, a motorcycle, etc. For example, the global device 110 may be a device that is part of the console of a car or a device in a seat back of an aircraft. The global device 110 may be described as an external or non-worn device (the global device 110 is not worn by a user (i.e., a person)).

[0013] The global device 110 may be a device integrated into a vehicle, and a global touchscreen 122 may be used to select various services (e.g., navigation, music, making phone calls, etc.). The global device 110 is coupled to or includes a data store 140. The global device 110 includes a global correction engine 120, a global touchscreen 122, and may optionally include a global motion sensor 124. Thus, in some embodiments, the global device 110 includes a global motion sensor 124, and in other embodiments, the global device 110 does not include a global motion sensor 124. The global touchscreen 122 may be an interactive keyboard or other user interface.

[0014] Each local device 150a,...,150n may be a smartphone, a wearable device (e.g., a smart watch), a tablet computer, a laptop computer, or any other device with a touch screen. Each local device 150a,...,150n includes a local correction engine 160a,...,160n, a local touch screen 162a,...,162n, a local motion sensor 164a,...,164n, and local device and user information 166a,...,166n. The local device and user information 166a,...,166n includes one or more attributes about the local device 150a,...,150n and one or more attributes about each user using the local device 150a,...,150n and the user marks provided. The local device information includes a local device identifier that identifies the local device 150a,...,150n, and may include other attributes related to the local device 150a,...,150n (e.g., type of device (smartphone or smartwatch), brand of device, etc.). The user information related to the user includes a user identifier (e.g., username), a user mark (e.g., fingerprint, word, symbol, etc.), and may include other attributes related to the user.

[0015] The data store 140 includes local device and user information 142a, ..., 142n corresponding to the local device and user information 166a, ..., 166n. When the global correction engine 120 receives a user mark, the global correction engine 120 uses the local device and user information 142a, ..., 142n to identify the user mark, identify the user who provided the user mark, and identify that user's local device 150a, ..., 150n. In one embodiment, the data store 140 includes local device and user information 142a, ..., 142n for a subset of the local devices 150a, ..., 150n (because one or more of the local devices 150a, ..., 150n have a configuration that does not communicate with the global device 110).

[0016] In one embodiment, each motion sensor 124, 164a, ..., 164n is an accelerometer. In one embodiment, any motion sensor 124, 164a, ..., 164n that provides device motion (e.g., acceleration) or that is used to calculate device motion (e.g., acceleration) may be used.

[0017] The global compensation engine 120 communicates with the local compensation engines 160a, ..., 160n to adjust location inputs (e.g., positions) on the global touchscreen 122. For example, if a user in a car intends to place a finger at location X50, Y20 on the global touchscreen 122 to select a "call" button, but due to the movement of the car, the finger is mistakenly placed at location X40, Y15, then the global compensation engine 120 identifies the correct location of X50, Y20 as the input location for the global touchscreen 122 and, in some embodiments, notifies the local compensation engines 160a, ..., 160n to accept the input of X50, Y20 when the local compensation engines 160a, ..., 160n are to perform certain processing based on the location input. In particular, the global compensation engine 120 leverages the local motion sensors 164a, ..., 164n of the local devices 150a, ..., 150n to make the compensation.

[0018] In an embodiment, the global correction engine 120 corrects for errors made by a driver of an automobile (i.e., a type of vehicle 100) while typing on the global touchscreen 122, where the errors result from sudden jolts transmitted to the driver as the motion of the automobile is affected by bumps, road conditions, braking, vibrations, etc. In this embodiment, the global correction engine 120 corrects for errors made by a passenger in an aircraft (a type of vehicle 100) using the global touchscreen 122 during turbulence. In these examples, conventional solutions that rely on accelerometers (a type of motion sensor) do not work well because touchscreen sensors deployed in automobiles or aircraft typically do not have accelerometers.

[0019] Even when an accelerometer is introduced into the car or airplane along with the global touchscreen 122, the passenger / driver and the global touchscreen 122 are not unique rigid bodies. Instead, dampers or shock absorbers introduced into the seat (in the car or airplane) or vehicle create more complex physical conditions where the global touchscreen 122 is not being held by a user who may be moving, which means that the user and the global touchscreen 122 are not unique rigid bodies.

[0020] Although examples herein may refer to automobiles, embodiments also apply to other vehicles (eg, aircraft).

[0021] FIG 2 illustrates devices in a car according to one embodiment. In FIG 2, a car 200 includes a global device 210 (i.e., a vehicle device) having a global correction engine 212 and a global touch screen 214. The car also includes two smartphones 230, 240 (i.e., local devices). The driver (one user) has one smartphone 230 with a local correction engine 232 and a local motion sensor 234, and a passenger (another user) has another smartphone 240 with a local correction engine 242 and a local motion sensor 234.

[0022] When the automobile 200 experiences an impact (e.g., due to a bump in the road, road conditions, braking, vibration, etc.), the relative movement of the global touchscreen 214 introduced into the automobile console will be different from the movement of each user (due to a number of factors, such as the seat absorbing part of the impact, the distance from the global touchscreen 214, the attributes of the user, etc.). This leads to an error proportional to the difference between the movement (e.g., acceleration) of the global touchscreen 214 and the movement (e.g., acceleration) of the user when the user touches the global touchscreen 214. In this example, the user and the global touchscreen 214 are not unique rigid bodies, which renders the (potential) use of any motion sensors (e.g., accelerometers) present in the automobile 200 ineffective.

[0023] Assuming that the user and smartphone 230 or 240 form one rigid body, the error is proportional to the difference in the motion (e.g., acceleration) of the two objects, global touchscreen 214 (object 1) versus smartphone 230, 240 (object 2). In an embodiment, the global compensation engine 212 of the vehicle device can communicate with the local compensation engines 232, 242 of the smartphones 230, 240 to obtain the motion of the smartphones 230, 240 from the local motion sensors 234, 244. The global compensation engine 212 then adjusts the position input of the user's finger on the global touchscreen 214 based on the actual motion of the user's smartphone 230, 240 (where the user's finger is designated to move in the same way as the smartphone 230, 240).

[0024] FIG. 3 illustrates device motion (e.g., acceleration) in a car according to an embodiment. In FIG. 3, a user has a smartphone 230 and is using a global touchscreen 214. If there is a bump in the road while the car 200 is moving, the user's seat causes the user to make a longer vertical movement 310 to better absorb the bump. However, the global touchscreen 214 follows the car's motion and has a shorter vertical movement 320. The user's finger movement is measured as Dp' 300 by a motion sensor on the user's smartphone 230. The global touchscreen 214 uses the measurement (Dp') from the smartphone 230 motion sensor as an input and corrects the position input of the user's finger on the global touchscreen 214.

[0025] FIG. 4 illustrates a global device and a local device according to an embodiment. The global device 400 is coupled to a database 410 (i.e., a type of data store) for storing local device information and user information. In an embodiment, the global correction engine of the global device 400 is a plug-in. The global device 400 can also recognize and communicate with the local devices 420, 430, and 440. The global device 400 has a global touchscreen 402 having a display surface 404. The global correction engine of the global device 400 receives sensor input data from a touch sensor of the global touchscreen 402 indicating position information relative to the display surface 404 of the touchscreen 402.

[0026] First, the global correction engine of the global device 400 discovers and registers the local devices 420, 430, 440. The global correction engine of the global device 400 also associates the local devices with a user based on a user mark (e.g., a fingerprint, a word, a symbol, etc.) provided on the local touchscreen of the local device 420, 430, 440. The global correction engine of the global device 400 then applies a correction factor to the finger movement on the global touchscreen based on the movement determined by the motion sensors of the local devices 420, 430, 440.

[0027] 5 is a flow chart illustrating operations for device registration according to one embodiment. Control begins at block 500 where the global correction engine 120 of the global device 110 communicates with each local correction engine 160a, ..., 160n of each local device 150a, ..., 150n in the vehicle 100 to request local device information 142. In one embodiment, the device information is a device identifier. In other embodiments, other device information may be provided, such as a protocol for communication.

[0028] In block 502, the local correction engines 160a, ..., 160n of each local device 150a, ..., 150n in the vehicle transmit the local device and user information 142a, ..., 142n to the global correction engine 120. In block 504, the global correction engine 120 receives the local device and user information 142a, ..., 142n from each local correction engine 160a, ..., 160n. In block 506, the global correction engine 120 stores the local device and user information 142a, ..., 142n in the data store 140.

[0029] 5 allows for binding the global device 110 and each local device 150a,...,150n. In various embodiments, the identification of the devices for binding them and exchanging information may occur in different ways.

[0030] 6A, 6B, and 6C illustrate registration, monitoring, and correction, according to one embodiment. Control begins at block 600, where the global correction engine 120 receives a first position input from a user touching a global touchscreen 122 of a global device 110 of a moving vehicle 100, where the position input is a position relative to a display surface of the global touchscreen 122 and indicates a user mark. That is, in one embodiment, the user uses a finger to provide a user mark, such as a fingerprint, at the location of the first position input. The process of block 600 may be described as receiving sensor input data from a touch sensor of the global touchscreen 122, where the sensor input data provides a position input relative to a display surface of the touchscreen.

[0031] In block 602, the global correction engine 120 determines whether user marks for the user are stored in the data store 140. If so, processing continues to block 604; otherwise, processing continues to block 620 (of FIG. 6C).

[0032] In block 604, the global correction engine 120 retrieves from the data store 140 the local device and user information 142a...142n associated with the user mark to identify the local devices 150a...150n of that user. In one embodiment, the user mark is a fingerprint, and the global correction engine 120 receives the user's fingerprint and uses the user's fingerprint to identify the local devices 150a...150n. In one embodiment, multiple users may access the same global touchscreen 122, and the local device and user information 142a...142n retrieved is specific to the user who received the user mark.

[0033] In block 606, based on the monitoring, the global compensation engine 120 determines that the movement of the global device 110 is different from the movement of the local devices 150a,...,150n. In block 608, the global compensation engine 120 determines whether the first position input needs to be adjusted based on the movement. If so, processing proceeds to block 610 (of FIG. 6B); otherwise, processing continues to block 600 (of FIG. 6A).

[0034] When determining whether to correct the error, the global correction engine 120 considers whether the local devices 150a, ..., 150n providing the motion information are one rigid body with the user. Sometimes the local devices 150a, ..., 150n are not one rigid body with the user. For example, this can happen when the local devices 150a, ..., 150n are on a bag or on a shelf of a vehicle (where the local devices 150a, ..., 150n are one rigid body with the vehicle and thus with the global touch screen 122). As another example, this can happen when the user is moving the local devices 150a, ..., 150n (e.g., with one hand while using the local devices 150a, ..., 150n with the other hand). Thus, the global correction engine 120 decides whether to apply a correction factor. If the local devices 150a,...,150n are one rigid body with the global touchscreen 122, the correction factor may be zero or near zero, which results in no or small correction being applied. If the local devices 150a,...,150n are moving, the correction factor may be either high or fluctuating over a short time frame, which may be the two conditions checked by the global correction engine 120 to decide whether or not to perform a correction.

[0035] In block 610, the global compensation engine 120 requests motion information (e.g., acceleration information) from the local compensation engines 160a, ..., 160n of the local devices 150a, ..., 150n, which is indicative of the motion of the user associated with the local devices 150a, ..., 150n. In block 612, the local compensation engines 160a, ..., 160n of the local devices 150a, ..., 150n obtain the motion information from the local motion sensors 164a, ..., 164n and transmit the motion information to the global compensation engine 120. In block 614, the global compensation engine 120 receives the motion information, which may be described as receiving motion information from the motion sensors 164a, ..., 164n of the local devices 150a, ..., 150n.

[0036] In block 616, the global correction engine 120 corrects the first position input based on the motion information to a second position input to be used as an input to the global touchscreen. For example, if the user places a finger on the global touchscreen 122 when the car goes over a bump, then the user has a different motion than the global touchscreen 122 and the user's finger lands at a location on the display surface of the global touchscreen 122 different from that desired by the user (e.g., on the "End Call" button on the global touchscreen 122). To correct for this, the global correction engine 120 takes into account the user's motion information to determine a second position input on the display surface of the global touchscreen 122 that is considered as the desired location on the global touchscreen 122 (e.g., on the "Mute" button on the global touchscreen 122). In block 618, in response to the second position input, the global device 110 performs an action (e.g., the second position input selects the "Mute" button and the global device 110 performs the mute function). Thus, by correcting the first location input to the second location input, in this example, the global device 110 processes a selection of the "Mute" button instead of a selection of the "End Call" button. From block 618 (of FIG. 6B), processing continues to block 600 (of FIG. 6A).

[0037] In block 620 (of FIG. 6C), the global correction engine 120 requests user information, including user marks, from the local correction engines 160a, ..., 160n of the user's local devices 150a, ..., 150n. In block 622, the local correction engines 160a, ..., 160n transmit user information about the user associated with the user marks and associated with the local devices. In various embodiments, the returned user information may be a portion (i.e., the user information portion) of the local devices and the user information 166a, ..., 166n, or may be all of the user information. In block 624, the global correction engine 120 stores the user information, including the user marks, in a data store. From block 624 (of FIG. 6C), processing continues to block 604 (of FIG. 6A). Thus, registration of a user occurs by the user providing user marks to be used to control the local devices 150a, ..., 150n. In such an embodiment, both the global correction engine 120 and the local correction engines 160a, ..., 160n can detect the user marks, and each of the local correction engines 160a, ..., 160n can transmit the user marks to the global correction engine 120.

[0038] The binding of the local devices 150a,...,150n to the global device 110 can be done in various ways. For example, in the case of a car driver, no new logic is needed since modern cars already allow the local devices 150a,...,150n (e.g., smartphones) to be registered for some functions. This may also be the case for aircraft, in some cases allowing the local devices 150a,...,150n to be used to listen to music provided by the aircraft. In addition, passengers or users of the aircraft touchscreen can register the local devices 150a,...,150n when boarding the vehicle or on the global touchscreen itself by binding them via Bluetooth® cord or other technology.

[0039] In certain embodiments, network delays in communication with / from the local devices 150a,...,150n and the global device 110 may not be fast enough for the immediate application of the two (vehicle and user) movements. In some embodiments, the corrections applied by the global correction engine 120 may occur a short time after the error is typed. However, in such embodiments, such near real-time corrections are sufficient so that the error is corrected before the user notices the error or the error affects the user's next decision.

[0040] In one embodiment, the global compensation engine 120 discovers how many local devices 150a, ..., 150n can interact with the global touchscreen 122 and who the user (e.g., owner) of each of the local devices 150a, ..., 150n is. Then, when the global touchscreen 122 receives a finger touch input, the global compensation engine 120 can associate the finger with a user and a local device 150a, ..., 150n and apply compensation for that user (based on local motion sensor information of the user's local devices 150a, ..., 150n).

[0041] In some embodiments, a user actively registers local devices 150a,...,150n for multiple reasons (e.g., to download music, to make calls from a centralized system, etc.). Additionally, the global correction engine 120 can detect if a user and local devices 150a,...,150n are registered when the user touches the global touch screen 122 with a user mark. If the user mark is not recognized (i.e., not found in the data store 140), the global correction engine 120 requests registration of the user.

[0042] In some embodiments, not all touchscreens recognize user marks and not all user marks are registered. In such embodiments, the user is asked to touch a limited area of ​​the touchscreen (global or local touchscreen) to register a user mark. In some embodiments where multiple users use the same global touchscreen 122, the global calibration engine 120 determines the calibration based on the local device 150a,...,150n associated with the last user to register a user mark.

[0043] Also, in some embodiments, devices are capable of finger recognition on all of their touchscreens. If a fingerprint touching the global touchscreen 122 has not been recognized, the global correction engine 120 may correct the position input using the last connected user (e.g., treating this as a good approximation) if such registration occurred within a reasonable time window in the past.

[0044] In one embodiment, the global calibration engine 120 performs the calibration in response to a user touching a finger on the global touch screen 122. The global calibration engine 120 then searches for the fingerprint in a data store and, if the fingerprint is not found, initiates enrollment of the user to obtain a fingerprint.

[0045] In other embodiments, the user provides a user mark (e.g., with a pen or other device used for input on the global touch screen 122). The global correction engine 120 then searches for the user mark in the data store, and if the fingerprint is not found, begins enrolling the user to obtain the user mark.

[0046] Then, each time a user touches the global touch screen 122 with a finger, the global compensation engine 120 obtains the relative movement of the local devices 150a, ..., 150n and applies compensation factors to compensate for the finger position input on the global touch screen 122.

[0047] In one embodiment, the global correction engine 120 determines the amount of change to the position input using the following formula: (Number 1) Change in position input Δp=1 / 2α local device Δt 2

[0048] That is, a change in position input (i.e., position) is calculated as the acceleration (α) of the local devices 150a,...,150n multiplied by the unit of time that the user acts on the global touchscreen (Δt 2 ) In one embodiment, the unit of time is the estimated unit of time.

[0049] In an embodiment, the local motion sensors 164a,...,164n may provide both magnitude and direction (latitude x, longitude y, and height z). In various embodiments, the global correction engine 120 considers one or more of the latitude x, longitude y, and height z (e.g., an embodiment may consider latitude x). In an embodiment, the global correction engine 120 estimates the change in position input for a very short period of time as the time difference between an acceleration event registered by the local motion sensors 164a,...,164n and the time the user presses the global touch screen 122.

[0050] In case of latency in communication between the local devices 150a,...,150n and the global touch screen 122, the correction is applied with some delay, which works for various use cases such as switching radio stations, configuring phone numbers, searching for contacts in an address book, etc. In this case, the global correction engine 120 determines the amount of change to the position input using the following formula: (Number 2) Change in position input Δp=1 / 2(α local device -α global device) Δt 2

[0051] That is, the change in position input (i.e., position) is the acceleration (α) about the local devices 150a,...,150n minus the acceleration (α) about the global touchscreen 122 over a unit of time (Δt 2 ) In one embodiment, the unit of time is the estimated unit of time.

[0052] The embodiments use the acceleration information coming from the local devices 150a, ..., 150n having local motion sensors 164a, ..., 164n associated with the users to calculate and mitigate errors when interacting with the global touchscreen 122. In the case of multiple potential users of one single global touchscreen 122, the embodiments identify which user is interacting with the global touchscreen 122 and select the correct source of motion information (local devices 150a, ..., 150n) in order to apply the correct corrections to position inputs to the global touchscreen 122.

[0053] In one embodiment, the motion is acceleration, and the embodiment calculates the relative acceleration and understands from which of the multiple local devices 150a,...,150n the other acceleration information is coming in. The embodiment links the user's own acceleration with the acceleration of the vehicle 100 via the local devices 150a,...,150n.

[0054] An embodiment processes a user's position input received from a touch sensor of a global touchscreen 122 of the global device 110. The embodiment receives sensor input data from the touch sensor, where the sensor input data includes position input providing position information relative to a display surface of the global touchscreen 122, receives motion information from local motion sensors 164a, ..., 164n of local devices 150a, ..., 150n, and corrects the position input based on the motion information. In an embodiment, the motion information is received from local motion sensors 164a, ..., 164n located on local devices 150a, ..., 150n remotely with respect to the global device 110.

[0055] In an embodiment, the global device 110 and the local devices 150a, ..., 150n are movable relative to each other. Motion information is received from the local devices 150a, ..., 150n, which are mobile computing devices carried by users. The local devices 150a, ..., 150n are registered with the global device 110. The global device 110 receives local device and user information 142a, ..., 142n identifying the registered users and selects local motion sensors 164a, ..., 164n of the local devices 150a, ..., 150n associated with the identified users. The global device 110 receives the motion information from the selected local motion sensors 164a, ..., 164n and corrects the position input of the sensor input data.

[0056] In an embodiment, the local device and user information 142a,...,142n includes a fingerprint of a user enrolled using the global touchscreen 122. The local motion sensors 164a,...,164n are operable to generate motion information describing motion, such as acceleration. The global device 110 may be a vehicle user interface device.

[0057] 7 illustrates a computing environment according to one embodiment. With reference to FIG. 7, computer system 712 is merely one example of a suitable computer system and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the present disclosure described herein. Nevertheless, computer system 712 is capable of implementing and / or performing any of the functionality described above.

[0058] Computing system 712 is a computer system capable of operation with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments or configurations or combinations thereof suitable for use with computer system 712 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, distributed cloud computing environments including any of the systems or devices described above, and the like.

[0059] The computer system 712 may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system 712 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0060] 7, a computer system 712 is shown in the form of a general purpose computing device. Components of the computer system 712 may include, but are not limited to, a processor or processing unit 716, a system memory 728, and a bus 718 that couples various system components including the one or more processors or processing units 716 to the system memory 728.

[0061] Bus 718 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or local bus, using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0062] Computer system 712 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system 712 and includes both volatile and nonvolatile media, removable and non-removable media.

[0063] The system memory 728 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 730 or cache memory 732, or both. The computer system 712 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 734 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, but commonly referred to as a "hard drive"). Not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such an example, each may be connected to the bus 718 by one or more data media interfaces. As shown and described in further detail below, the system memory 728 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present disclosure.

[0064] Programs / utilities 740 having a set (at least one) of program modules 742 may be stored in system memory 728, by way of example and not limitation, as well as an operating system, one or more application programs, other program modules and program data. Each of the operating system, one or more application programs, other program modules and program data, or any combination thereof, may include an implementation of a network environment. The program modules 742 generally perform the functions and / or methodologies of the embodiments described herein.

[0065] The computer system 712 may communicate with one or more external devices 714, such as a keyboard, pointing device, display 724; one or more devices that allow a user to interact with the computer system 712; or any device (e.g., a network card, modem, etc.) that allows the computer system 712 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur through an input / output (I / O) interface 722. Additionally, the computer system 712 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, through a network adapter 720. As shown, the network adapter 720 communicates with other components of the computer system 712 through a bus 718. It is understood that other hardware and / or software components, not shown, may be used with the computer system 712. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives and data archive storage systems.

[0066] In one embodiment, the global computing device 110 and each local computing device 150...150n has a computer system 712 architecture.

[0067] Additional Embodiments The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail. The computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0068] A computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. Computer-readable storage media, as used herein, is not to be construed as a transitory signal per se, such as an electric wave, a freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0069] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.

[0070] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented languages ​​such as Smalltalk®, C++ or the like, procedural languages ​​such as the C programming language or similar programming languages. The computer readable program instructions may be executed as a stand-alone software package, entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, the electrical circuitry may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electrical circuitry, which may include, for example, programmable logic circuitry, a field programmable gate array (FPGA), or a programmable logic array (PLA), to perform aspects of the invention.

[0071] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer readable program instructions.

[0072] These computer readable program instructions, when provided to a computer processor or other programmable data processing apparatus, cause the instructions, executed via the computer processor or other programmable data processing apparatus, to create a machine that creates means for implementing the functions / acts identified in the block or blocks of the flowchart illustrations and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus or other device or combination thereof to function in a particular manner, such that a computer readable storage medium having instructions stored thereon includes an article of manufacture including instructions that implement aspects of the functions / acts identified in the block or blocks of the flowchart illustrations and / or block diagrams.

[0073] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to generate a computer-implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement aspects of the functions / actions identified in a block or blocks of the flowchart illustrations or / and block diagrams.

[0074] The flowchart diagrams and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a particular logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order depicted in the drawings. For example, two blocks shown in succession may in fact be realized as a single step, or may be executed simultaneously, substantially simultaneously, partially or wholly in a temporally overlapping manner, or the blocks may be executed in the reverse order depending on the functionality involved. It should be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special purpose hardware-based system that performs a particular function or action, or that implements a combination of special purpose hardware and computer instructions.

[0075] "A embodiment," "embodiment," "embodiments," "the embodiment," "the embodiment," "the embodiments," "one or more embodiments," "some embodiments," and "one embodiment" refer to "one or more (but not all) embodiments" of the present invention, unless expressly specified otherwise.

[0076] The terms "including," "comprising," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.

[0077] An enumerated listing of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise.

[0078] The terms "a," "an," and "the" mean "one or more," unless expressly specified otherwise.

[0079] Devices that are in communication with each other need not communicate with each other continuously, unless otherwise explicitly specified, In addition, devices that are in communication with each other may communicate directly or indirectly through one or more intermediate stages.

[0080] A description of an embodiment with several components in communication with each other does not imply that all such components are required, on the contrary, a variety of optional components are described to illustrate various potential embodiments of the present invention.

[0081] It will be readily apparent that where a single device or product is described herein, one or more devices / products (whether in cooperation or not) may be used in place of the single device / product. Similarly, where more than one device or product is described herein (whether in cooperation or not), it will be readily apparent that a single device / product may be used in place of the more than one device or product, or a different number of devices / products may be used in place of the number of devices or programs illustrated. The functionality or features or both of a device may alternatively be embodied by one or more other devices not explicitly described as having such functionality / features. Thus, other embodiments of the invention may not include a device per se.

[0082] The above description of various embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Numerous modifications and variations are possible in light of the above teachings. It is intended that the scope of the invention be limited not by this detailed description, but rather by the appended claims. The above specification, examples and data provide a complete description of the manufacture and use of the composition of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, embodiments of the invention reside in the claims appended heretoafter. The above description provides examples of embodiments of the invention, and modifications and substitutions may be made in other embodiments.

Claims

1. 1. A computer-implemented method, comprising: discovering one or more local devices; receiving a first position input of a user touching a global touchscreen of a global device, the first position input being a position relative to a display surface of the global touchscreen and including information indicative of a user mark; identifying a local device of the one or more local devices that is associated with the user mark; determining a global device movement that differs from the identified local device movement; acquiring motion information from the motion sensor of the identified local device, the identified local device having a motion sensor; correcting the first position input to a second position input based on the movement information; performing an action on the global device based on the second position input; and 4. A computer-implemented method comprising:

2. An operation of receiving a user mark from each local device of the plurality of local devices; storing the user mark for each local device of the plurality of local devices; 2. The computer-implemented method of claim 1, wherein the user mark associated with the first position input corresponds to a stored user mark associated with one of the plurality of local devices.

3. The computer-implemented method of claim 1 or 2, wherein the local device is remote from the global device.

4. An operation of registering the one or more local devices to the global device. The computer-implemented method of any one of claims 1 to 3, comprising:

5. The computer-implemented method of any one of claims 1 to 4, wherein the user mark comprises a fingerprint.

6. The computer-implemented method of claim 1 , wherein the motion information includes acceleration.

7. The computer-implemented method of any one of claims 1 to 6, wherein the global touchscreen comprises a user interface device of a moving vehicle or aircraft.

8. A computer program comprising: discovering one or more local devices; receiving a first position input of a user touching a global touchscreen of a global device, the first position input being a position relative to a display surface of the global touchscreen and including information indicative of a user mark; identifying a local device of the one or more local devices that is associated with the user mark; determining a global device movement that differs from the identified local device movement; acquiring motion information from the motion sensor of the identified local device, the identified local device having a motion sensor; correcting the first position input to a second position input based on the movement information; performing an action on the global device based on the second position input; and A computer program for executing the above.

9. The computer includes: receiving a user mark from each local device of the plurality of local devices; storing the user mark for each local device of the plurality of local devices; 9. The computer program product of claim 8, further comprising: wherein the user mark associated with the first position input corresponds to a user mark stored for one of the plurality of local devices.

10. 10. The computer program product of claim 8 or 9, wherein the local device is remote from the global device.

11. The computer includes: registering said one or more local devices with said global device; The computer program product according to any one of claims 8 to 10, further comprising:

12. The computer program product of any one of claims 8 to 11, wherein the user mark comprises a fingerprint.

13. The computer program product of any one of claims 8 to 12, wherein the movement information includes acceleration.

14. The computer program product of any one of claims 8 to 13, wherein the global touchscreen comprises a user interface device of a moving vehicle or aircraft.

15. 1. A computer system comprising: one or more processors; one or more computer readable memories; one or more computer readable tangible storage devices; program instructions stored in at least one of said one or more computer-readable tangible storage devices for execution by at least one of said one or more processors via at least one of said one or more computer-readable memories to perform operations; and the operation includes: discovering one or more local devices; receiving a first position input of a user touching a global touchscreen of a global device, the first position input being a position relative to a display surface of the global touchscreen and including information indicative of a user mark; identifying a local device of the one or more local devices that is associated with the user mark; determining a global device movement that differs from the identified local device movement; acquiring motion information from the motion sensor of the identified local device, the identified local device having a motion sensor; correcting the first position input to a second position input based on the movement information; performing an action on the global device based on the second position input; and 1. A computer system comprising:

16. The method of claim 1, further comprising: receiving a user mark from each of the plurality of local devices; storing the user mark for each local device of the plurality of local devices; 16. The computer system of claim 15, wherein a user mark associated with the first location input corresponds to a stored user mark associated with any of the stored plurality of local devices.

17. 17. The computer system of claim 15 or 16, wherein the local device is remote from the global device.

18. The operation of registering the one or more local devices to the global device. The computer system according to any one of claims 15 to 17, comprising:

19. The computer system of any one of claims 15 to 18, wherein the user mark comprises a fingerprint.

20. The computer system of any one of claims 15 to 19, wherein the movement information includes acceleration.

Images