Automatically correcting touch screen errors

Abstract

A technique for automatically correcting touchscreen errors is provided. A first position input, where the first position input is a position relative to the display surface of the global touchscreen, is received from a user touching the global touchscreen of a global device. It is determined that movement of the global device differs from movement of local devices. Motion information of the local devices is retrieved. Based on the motion information, the first position input is corrected to a second position input. An operation is performed in response to the second position input.

Description

Technical Field

[0001] Embodiments of the present invention relate to automatically correcting touchscreen errors. Specifically, embodiments of the present invention relate to automatically correcting touchscreen errors of a device using an external or unworn information source. Background Technology

[0002] There are strategies and systems in place to detect and correct erroneous typing when using a device's touchscreen. Some strategies are based on human behavior and general interaction with the touchscreen (especially its keyboard), while others are based on motion sensors (e.g., accelerometers).

[0003] These strategies are useful when detecting contextual impairments by utilizing motion sensors within the device that expose the touchscreen. These strategies are based on the assumption that the relative movement of the finger and the touchscreen is identical, such that errors caused by walking vibrations and / or loss of attention can be corrected by the device's internal motion sensors.

[0004] For example, a person is holding a smartphone with a touchscreen and walking. The motion sensor inside the smartphone measures the person's acceleration based on the assumption that the smartphone and the person are a single rigid body. Therefore, the inertia that causes the relative incremental (delta) movement of the finger on the touchscreen is proportional to the acceleration measured by the smartphone's motion sensor. Summary of the Invention

[0005] According to some embodiments, a computer-implemented method for automatically correcting touchscreen errors is provided. The computer-implemented method includes operations: receiving a first position input from a user touching a global touchscreen of a global device, wherein the first position input is a position relative to the display surface of the global touchscreen; determining that movement of the global device differs from movement of local devices; retrieving motion information of the local devices; correcting the first position input to a second position input based on the motion information; and performing operations in response to the second position input.

[0006] According to other embodiments, a computer program product for automatically correcting touchscreen errors is provided. The computer program product includes a computer-readable storage medium containing program code executable by at least one processor to perform operations. A first position input is received from a user touching a global touchscreen of a global device, wherein the first position input is a position relative to the display surface of the global touchscreen. It is determined that movement of the global device differs from movement of local devices. Motion information of the local devices is retrieved. The first position input is corrected to a second position input based on the motion information. Operations are performed in response to the second position input.

[0007] According to other embodiments, a computer system for automatically correcting touchscreen errors is provided. The computer system includes: one or more processors, one or more computer-readable storage devices, and one or more computer-readable tangible storage devices; and program instructions stored on at least one of the one or more computer-readable tangible storage devices, for execution by at least one of the one or more processors via at least one of the one or more memories to perform operations. A first position input is received from a user touching a global touchscreen of a global device, wherein the first position input is a position relative to the display surface of the global touchscreen. It is determined that movement of the global device differs from movement of a local device. Motion information of the local device is retrieved. The first position input is corrected to a second position input based on the motion information. Operations are performed in response to the second position input. Attached Figure Description

[0008] Referring now to the accompanying drawings, in which the same reference numerals always denote corresponding parts:

[0009] Figure 1 A computing environment according to certain embodiments is illustrated in block diagram.

[0010] Figure 2 A device in a car according to certain embodiments is shown.

[0011] Figure 3 The movement of a device in a car according to certain embodiments is shown.

[0012] Figure 4 Global and local devices according to certain embodiments are shown.

[0013] Figure 5 The flowchart illustrates the operations for device registration according to certain embodiments.

[0014] Figure 6A , Figure 6B and Figure 6C Registration, monitoring, and correction are illustrated according to certain embodiments.

[0015] Figure 7 A computing environment according to certain embodiments is shown. Detailed Implementation

[0016] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0017] Examples of embodiments automatically correct for typing errors on the touchscreen of devices that lack motion sensors (e.g., accelerometers) or whose internal motion sensors cannot be directly correlated with user movements. For example, embodiments automatically correct touchscreen errors in the following scenarios:

[0018] -The touchscreen and the user using the touchscreen are not a rigid body; therefore, the relative movement of the user's finger and the touchscreen is not the same.

[0019] - The device does not have an installed motion sensor; and

[0020] - More than one user may use the same touchscreen at the same time.

[0021] Figure 1 A computing environment according to certain embodiments is illustrated in block diagram. Vehicle 100 includes a global device 110 and one or more local devices 150a...150n. Vehicle 100 may be a car, electric bicycle, airplane, truck, motorcycle, etc. For example, global device 110 may be a device that is part of the console of a car or may be a device on the back of a seat in an airplane. Global device 110 may be described as an external or non-wearable device (e.g., global device 110 is not worn by a user (i.e., a person)).

[0022] Global device 110 may be a vehicle-integrated device, and global touchscreen 122 may be used to select various services (e.g., navigation, music, phone calls, etc.). Global device 110 is coupled to or includes data repository 140. Global device 110 includes global calibration engine 120, global touchscreen 122, and may optionally include global motion sensor 124. Thus, in some embodiments, global device 110 includes global motion sensor 124, and in other embodiments, global device 110 does not include global motion sensor 124. Global touchscreen 122 may be an interactive keyboard or other user interface.

[0023] Each local device 150a…150n can be a smartphone, wearable device (e.g., a smartwatch), tablet computer, laptop computer, or any other device with a touchscreen. Each local device 150a…150n includes a local calibration engine 160a…160n, a local touchscreen 162a…162n, a local motion sensor 164a…164n, and local device and user information 166a…166n. The local device and user information 166a…166n includes one or more attributes about the local device 150a…150n and one or more attributes about each user using the local device 150a…150n, as well as provided user tags. The local device information includes a local device identifier that identifies the local device 150a…150n, and may include other attributes about the local device 150a…150n (e.g., device type (smartphone or smartwatch), device brand, etc.). The user's user information includes a user identifier (e.g., username), user tags (e.g., fingerprint, words, symbols, etc.), and may include other attributes about the user.

[0024] Data repository 140 includes local device and user information 142a…142n corresponding to local devices and user information 166a…166n. When global correction engine 120 receives a user tag, it uses the local device and user information 142a…142n to locate the user tag, identify the user who provided the user tag, and the local device 150a…150n that identifies the user. In some embodiments, data repository 140 includes local device and user information 142a…142n for a subset of local devices 150a…150n (e.g., because one or more local devices 150a…150n have settings that prevent them from communicating with global device 110).

[0025] In some embodiments, each motion sensor 124, 164a…164n is an accelerometer. In some embodiments, any motion sensor 124, 164a…164n that provides motion (e.g., acceleration) of the device or can be used to calculate motion (e.g., acceleration) of the device can be used.

[0026] The global calibration engine 120 communicates with the local calibration engines 150a...150n to adjust the position input (e.g., location) on the global touchscreen 122. For example, if a user in a car wants to place their finger on positions X50, Y20 on the global touchscreen 122 to select the "call" button, but due to the movement of the car, the finger is unintentionally placed on positions X40, Y15, the global calibration engine 120 recognizes the correct position of X50, Y20 as the position input to the global touchscreen 122. In some embodiments, when the local calibration engines 160a...160n are about to perform some processing based on the position input, the global calibration engine 120 notifies the local calibration engines 160a...160n to accept the input of X50, Y20. In particular, the global calibration engine 120 utilizes the local motion sensors 164a...164n of the local devices 150a...150n for calibration.

[0027] In one embodiment, the global correction engine 120 corrects errors made by a driver of a car (i.e., a type of vehicle 100) when typing on a global touchscreen 122, where such errors occur due to sudden vibrations transmitted to the driver when the car's movement is affected by bumps, road conditions, braking, vibrations, etc. In another embodiment, the global correction engine 120 corrects errors made by passengers of an aircraft (i.e., a type of vehicle 100) using the global touchscreen 122 during turbulent conditions. In these examples, conventional solutions relying on accelerometers (a type of motion sensor) do not work because touchscreens installed in cars or aircraft typically do not have accelerometers.

[0028] Even if an accelerometer is installed in a car or aircraft with a global touchscreen 122, the passenger / driver and the global touchscreen 122 are not the only rigid bodies. Instead, dampers or shock absorbers installed in the seats or vehicles (of a car or aircraft) create a more complex physical situation where the global touchscreen 122 is not held by a user who may be moving, meaning that the user and the global touchscreen 122 are not the only rigid bodies.

[0029] While the examples in this article may relate to automobiles, the embodiments are also applicable to other vehicles (e.g., airplanes).

[0030] Figure 2 A device in a car according to certain embodiments is shown. Figure 2In the vehicle 200, there is a global device 210 (i.e., vehicle device) having a global correction engine 212 and a global touchscreen 214. The vehicle also includes two smartphones 230 and 240 (i.e., local devices). The driver (one user) has a smartphone 230 with a local correction engine 232 and a local motion sensor 234, and the passenger (another user) has a smartphone 240 with a local correction engine 242 and a local motion sensor 234.

[0031] If the car 200 experiences vibrations (e.g., due to bumps in the road, road conditions, braking, vibrations, etc.), the relative movement of the global touchscreen 214 mounted on the car's console differs from the movement of each user (due to numerous factors such as seats that absorb some vibrations, distance from the global touchscreen 214, user attributes, etc.). This results in an error when the user touches the global touchscreen 214, which is proportional to the difference between the motion of the global touchscreen 214 (e.g., acceleration) and the user's motion (e.g., acceleration). In this example, the user and the global touchscreen 214 are not the only rigid bodies, and this renders the (potential) use of any motion sensors (e.g., accelerometers) present in the car 200 ineffective.

[0032] Assuming the user and smartphone 230 or 240 form a rigid body, the error is proportional to the difference in motion (e.g., acceleration) between the global touchscreen 214 (object 1) and smartphone 230, 240 (object 2). In an embodiment, the vehicle device's global correction engine 212 is capable of communicating with the local correction engines 232, 242 of smartphone 230, 240 to obtain the motion of smartphone 230, 240 from local motion sensors 234, 244. The global correction engine 212 then adjusts the user's finger position input on the global touchscreen 214 based on the actual motion of the user's smartphone 230, 240 (wherein the user's finger is designated to move in the same manner as smartphone 230, 240).

[0033] Figure 3 The movement (e.g., acceleration) of a device in a car according to certain embodiments is illustrated. Figure 3In this scenario, a user has a smartphone 230 and is using a global touchscreen 214. When the car 200 is moving and there are bumps on the road, the user's seat allows for a longer vertical movement 310 to better absorb these bumps. However, the global touchscreen 214 follows the car's movement and has a shorter vertical movement 320. The user's finger movement is measured as Dp' 300 by a motion sensor on the user's smartphone 230. The global touchscreen 214 uses the measurement (Dp') from the smartphone 230's motion sensor as input and corrects for the user's finger position input on the global touchscreen 214.

[0034] Figure 4 A global device and local devices according to certain embodiments are illustrated. Global device 400 is coupled to a database 410 (i.e., a type of data repository) for storing local device information and user information. In some embodiments, the global calibration engine of global device 400 is a plug-in. Global device 400 also identifies local devices 420, 430, and 440 and is capable of communicating with them. Global device 400 has a global touchscreen 400 with a display surface 404. The global calibration engine of global device 400 receives sensor input data from the touch sensor of global touchscreen 402, which indicates positional information relative to the display surface 404 of touchscreen 402.

[0035] Initially, the global calibration engine of global device 100 discovers and registers local devices 420, 430, and 440. The global calibration engine of global device 100 also associates local devices with users based on user identifiers (e.g., fingerprints, words, symbols, etc.) provided on the local touchscreens of local devices 420, 430, and 440. Then, the global calibration engine of global device 100 applies calibration factors to finger movements on the global touchscreen based on movements determined by the motion sensors of local devices 420, 430, and 440.

[0036] Figure 5 A flowchart illustrates operations for device registration according to certain embodiments. Control begins at block 500, where the global correction engine 120 of global device 110 communicates with each local correction engine 160a…160n of each local device 150a…150n in vehicle 100 to request local device information 142. In some embodiments, the device information is a device identifier. In other embodiments, other device information may be provided, such as protocols for communication.

[0037] In block 502, the local correction engines 160a…160n of each local device 150a…150n in the vehicle send local device and user information 142a…142n to the global correction engine 120. In block 504, the global correction engine 120 receives local device and user information 142a…142n from each local correction engine 160a…160n. In block 506, the global correction engine 120 stores the local device and user information 142a…142n in the data repository 140.

[0038] therefore, Figure 5 The processing allows for the coupling of global device 110 and each local device 150a...150n. In various embodiments, the identification of devices for coupling devices and exchanging information can be performed in different ways.

[0039] Figure 6A , Figure 6B and Figure 6C Registration, monitoring, and calibration are illustrated according to certain embodiments. Control begins at block 600, where the global calibration engine 120 receives a first location input from a user touching the global touchscreen 122 of the global device 110 of the mobile vehicle 100. This location input is a position relative to the display surface of the global touchscreen 122 and indicates a user mark. That is, in some embodiments, the user provides a user mark, such as a fingerprint, at the location of the first location input. The processing at block 600 can be described as receiving sensor input data from the touch sensor of the global touchscreen 122, where the sensor input data provides a location input relative to the display surface of the touchscreen.

[0040] In block 602, the global correction engine 120 determines whether the user's user tag is stored in the data repository 140. If yes, processing continues to block 604; otherwise, processing continues to block 620. Figure 6C ).

[0041] In block 604, the global calibration engine 120 retrieves local device and user information 142a...142n associated with a user tag from the data repository 140 to identify the user's local device 150a...150n. In some embodiments, the user tag is a fingerprint, and the global calibration engine 120 receives the user's fingerprint and uses the user's fingerprint to identify the local device 150a...150n. In some embodiments, multiple users can access the same global touchscreen 122, so the retrieved local device and user information 142a...142n is specific to the user whose user tag has been received.

[0042] In block 606, based on monitoring, the global correction engine 120 determines that the movement of the global device 110 is different from the movement of the local devices 150a...150n. In block 608, the global correction engine 120 determines, based on the movement, whether the first position input needs adjustment. If so, processing continues to block 610. Figure 6B Otherwise, processing continues to box 600. Figure 6A ).

[0043] When determining whether to correct for errors, the global correction engine 120 considers whether the local devices 150a…150n providing motion information are a rigid body with the user. Sometimes, the local devices 150a…150n are not a rigid body with the user. This can happen, for example, if the local devices 150a…150n are on a bag or on a shelf in a vehicle (in which case, the local devices 150a…150n are a rigid body with the vehicle, and therefore with the global touchscreen 122). As another example, this can happen if the user is moving the local devices 150a…150n (e.g., moving the local devices 150a…150n with one hand while using the local devices 150a…150n with the other hand). Therefore, the global correction engine 120 determines whether to apply a correction factor. In the case where the local devices 150a…150n are a rigid body with the global touchscreen 122, the correction factor can be zero or close to zero, resulting in no correction or a small correction being applied. When local devices 150a…150n are moving, the correction factor may fluctuate or be high within a short frame, which can be checked by the global correction engine 120 to determine whether to perform correction.

[0044] In block 610, the global calibration engine 120 requests motion information (e.g., acceleration information) from the local calibration engines 160a…160n of the local devices 150a…150n. This indicates the movement of the user associated with the local devices 150a…150n. In block 612, the local calibration engines 160a…160n of the local devices 150a…150n obtain motion information from the local motion sensors 164a…164n and send the motion information to the global calibration engine 120. In block 614, the global calibration engine 120 receives the motion information. This can be described as receiving motion information from the motion sensors 164a…164n of the local devices 162a…162n.

[0045] In block 616, global calibration engine 120 corrects a first position input to a second position input to be used as an input to the global touchscreen based on motion information. For example, if a user places their finger on global touchscreen 122 while a car is going over a bump, the user has a different motion than global touchscreen 122, and the user's finger lands on a different position on the display surface of global touchscreen 122 than the position the user expects (e.g., on the "End Call" button on global touchscreen 122). To correct for this, global calibration engine 120 considers the user's motion information to determine a second position input on the display surface of global touchscreen 122 that is considered the expected position on global touchscreen 122 (e.g., on the "Mute" button on global touchscreen 122). In block 618, in response to the second position input, global device 110 performs an operation (e.g., the second position input selects the "Mute" button, and global device 110 performs the mute function). Thus, by correcting the first position input to the second position input, in this example, global device 110 processes the selection of the "Mute" button instead of the selection of the "End Call" button. From box 618 ( Figure 6B Processing continues to box 600 ( Figure 6A ).

[0046] In box 620 ( Figure 6C In block 622, the global correction engine 120 requests user information, including user tags, from the local correction engines 160a…160n of the user's local devices 150a…150n. In block 622, the local correction engines 160a…160n send user information associated with the user tags and the local devices. In various embodiments, the returned user information may be a portion (i.e., a portion of the user information) or all of the local devices and user information 166a…166n. In block 624, the global correction engine 120 stores the user information, including user tags, in a data repository. From block 624… Figure 6C Processing continues to box 604. Figure 6A Therefore, user registration is performed by the user providing a user tag for controlling local devices 150a...150n. In such an embodiment, both the global correction engine 120 and the local correction engines 160a...160n are capable of detecting the user tag, and each local correction engine 160a...160n is capable of sending the user tag to the global correction engine 120.

[0047] The coupling between local devices 150a…150n and global device 110 can be performed in various ways. For example, in the case of a car driver, no new logic is required because modern cars already allow local devices 150a…150n (e.g., smartphones) to register several functions. In some cases, this may also apply to aircraft, allowing the use of local devices 150a…150n to listen to music provided by the aircraft. Furthermore, by coupling local devices 150a…150n via Bluetooth codes or other technologies, passengers or aircraft touchscreen users can register upon entering the vehicle or on the global touchscreen itself.

[0048] In some embodiments, network latency for communication to / from local devices 150a…150n and global device 110 may not be as fast as the instantaneous application of the two movements (of the vehicle and the user). In some embodiments, corrections applied by the global correction engine 120 may occur within a short period of time after the error has been entered. However, in such embodiments, such near real-time correction is sufficient because the error is corrected before the user notices it or before it affects the user's next decision.

[0049] In some embodiments, the global calibration engine 120 determines how many local devices 150a…150n (e.g., smartphones) can interact with the global touchscreen 122 and who is the user (e.g., owner) of each local device 150a…150n. Then, when the global touchscreen 122 receives finger touch input, the global calibration engine 120 can associate the finger with the user and the local device 150a…150n to apply calibration to that user (based on local motion sensor information of the user's local devices 150a…150n).

[0050] In some embodiments, users actively register local devices 150a...150n (e.g., smartphones) for various reasons (e.g., downloading music, making phone calls from a centralized system, etc.). Furthermore, when a user touches the global touchscreen 122 using a user tag, the global calibration engine 120 can detect whether the user and local devices 150a...150n have already registered. If the user tag is not recognized (i.e., not found in the data repository 140), the global calibration engine 120 requests the user to register.

[0051] In some embodiments, not all touchscreens recognize user tags, and not all user tags are registered. In such embodiments, the user is required to touch a limited area of ​​the touchscreen (global touchscreen or partial touchscreen) to register the user tag. In some embodiments, when multiple users use the same global touchscreen 122, the global calibration engine 120 determines calibration based on the local devices 150a…150n associated with the last user to register the user tag.

[0052] Furthermore, in some embodiments, there are devices where all touchscreens are capable of finger recognition. If a fingerprint touching the global touchscreen 122 is not recognized, the global correction engine 120 can still use the last connected user to correct the position input (e.g., consider this a good approximation) if such registration occurred within a reasonable time window in the past.

[0053] In some embodiments, the global calibration system 120 performs calibration in response to a user touching the global touchscreen 122 with a finger. The global calibration engine 120 then searches for fingerprints in a data repository, and if no fingerprint is found, it initiates user registration to obtain a fingerprint.

[0054] In other embodiments, the user provides a user token (e.g., using a pen or other device for input on the global touchscreen 122). The global calibration engine 120 then searches for the user token in a data repository, and if no fingerprint is found, user registration begins to obtain a user token.

[0055] Then, each time the user touches the global touchscreen 122 with their finger, the global correction engine 120 obtains the relative motion of the local devices 150a...150n and applies a correction factor to correct the position input of the finger on the global touchscreen 122.

[0056] In some embodiments, the global correction engine 120 uses the following formula to determine the amount of change in the position input:

[0057] The change in position input Δp = 1 / 2α 局部设备 Δt 2

[0058] That is, the change in position input (i.e., position) is equal to half the acceleration (α) of the local device 150a...150n multiplied by the time unit (Δt) of the user's operation on the global touchscreen. 2 In some embodiments, the time unit is an estimated time unit.

[0059] In embodiments, local motion sensors 164a…164n are capable of providing both the intensity and direction of movement (latitude x, longitude y, and altitude z). In various embodiments, the global correction engine 120 considers one or more of latitude x, longitude y, and altitude z (e.g., some embodiments may consider latitude x). In embodiments, the global correction engine 120 estimates changes in position input as a very small time difference between the acceleration event recorded by the local motion sensors 164a…164n and the time the user presses the global touchscreen 122.

[0060] In cases where there is a time delay in the communication between the local devices 150a…150n and the global touchscreen 122, correction can be applied with some delay. This works in various use cases, such as switching radio stations, composing phone numbers, and looking up contacts in an address book. In this case, the global correction engine 120 uses the following formula to determine the amount of change in location input:

[0061] The change in position input Δp = 1 / 2(α) 局部设备 –α 全局触摸屏 )Δt 2

[0062] That is, the change in position input (i.e., position) is equal to the acceleration (α) of the local device 150a...150n minus half the acceleration (α) of the global touchscreen 122 multiplied by the time unit (Δt) of the user's operation on the touchscreen. 2 In some embodiments, the time unit is an estimated time unit.

[0063] The embodiments calculate and mitigate errors during interaction with the global touchscreen 122 by using acceleration information from local devices 150a…150n, each having a local motion sensor 164a…164n associated with a user. In the case of multiple potential users on a single global touchscreen 122, the embodiments identify which user is interacting with the global touchscreen 122 and select the correct source of motion information (local devices 150a…150n) to apply the correct correction to position input to the global touchscreen 122.

[0064] In some embodiments, motion is acceleration, and the embodiments calculate relative acceleration and understand which of the multiple local devices 150a...150n other acceleration information comes from. The embodiments link the user's own acceleration to the acceleration of the vehicle 100 via local devices 150a...150n.

[0065] The embodiment processes user position input received from a touch sensor of a global touchscreen 122 of a global device 110. The embodiment receives sensor input data from the touch sensor, wherein the sensor input data includes position input providing position information relative to the display surface of the global touchscreen 122; receives motion information from position motion sensors 164a…164n of local devices 150a…150n; and corrects the position input based on the motion information. In the embodiment, motion information is received from local motion sensors 164a…164n located remotely relative to the global device 110 at local devices 150a…150n.

[0066] In this embodiment, the global device 110 and the local devices 150a…150n are movable relative to each other. Motion information is received from the local devices 150a…150n, which are mobile computing devices carried by the user. The local devices 150a…150n register with the global device 110. The global device 110 receives local device and user information 142a…142n that identifies the registered user, and selects local motion sensors 164a…164n of the local devices 150a…150n associated with the identified user. The global device 110 receives motion information from the selected local motion sensors 164a…164n and corrects the position input of the sensor input data.

[0067] In this embodiment, local devices and user information 142a…142n include the fingerprints of registered users using the global touchscreen 122. Local motion sensors 164a…164n are operable to generate motion information describing motion such as acceleration. The global device 110 may be a vehicle user interface device.

[0068] Figure 7 A computing environment according to certain embodiments is illustrated. References Figure 7 The computer system 712 is merely an example of a suitable computing system and is not intended to impose any limitation on the scope or functionality of the embodiments of the invention described herein. In any case, the computer system 712 is capable of implementing and / or performing any of the functions set forth above.

[0069] Computer system 712 can be a computer system that operates with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with computer system 712 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0070] Computer system 712 can be described within the general context of computer system executable instructions (such as program modules) executed by the computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer system 712 can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside on both local and remote computer system storage media, including memory storage devices.

[0071] like Figure 7 As shown, a computer system 712 is illustrated in the form of a general-purpose computing device. Components of the computer system 712 may include, but are not limited to, one or more processors or processing units 716, system memory 728, and a bus 718 that couples various system components, including system memory 728, to one or more processors or processing units 716.

[0072] Bus 718 represents one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. As an example and not a limitation, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0073] Computer system 712 typically includes various computer system readable media. Such media can be any available media accessible by computer system 712, and the media includes volatile and non-volatile media, removable and non-removable media.

[0074] System memory 728 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 730 and / or cache memory 732. Computer system 712 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 734 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (not shown, and generally referred to as a "hard disk drive"). Although not shown, a disk drive may be provided for reading from or writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media). In such a case, each may be connected to bus 718 via one or more data media interfaces. As will be further described and depicted below, system memory 728 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0075] A program / utility 740 having a set (at least one) of program modules 742, along with an operating system, one or more applications, other program modules, and program data, may be stored in system memory 728 in an exemplary, and not limited, manner. Each or some combination of the operating system, one or more applications, other program modules, and program data may include an implementation of a networking environment. Program modules 742 typically perform functions and / or methods as described herein in embodiments of the invention.

[0076] Computer system 712 can also communicate with one or more external devices 714 (e.g., keyboard, pointing device, display 724, etc.); one or more devices that enable a user to communicate with computer system 712; and / or any device that enables computer system 712 to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). Such communication can be performed via input / output (I / O) interface 722. Furthermore, computer system 712 can communicate with one or more networks such as a local area network (LAN), a general area network (WAN), and / or a public network (e.g., the Internet) via network adapter 720. As depicted, network adapter 720 communicates with other components of computer system 712 via bus 718. It should be understood that, although not shown, other hardware and / or software components can be used in conjunction with computer system 712. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0077] In some embodiments, the global computing device 100 and each local computing device 150...150n have the architecture of a computer system 712.

[0078] Additional Implementation Details

[0079] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0080] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices on which instructions are recorded (such as punched cards or raised structures in slots), and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0081] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0082] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet provided by an Internet service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by personalizing the electronic circuits with state information utilizing the computer-readable program instructions in order to perform aspects of this invention.

[0083] This document describes aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0084] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that instructs a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0085] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may not be performed in the order indicated in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0087] Unless otherwise expressly stated, the terms “an embodiment,” “embodiment,” “embodiments,” “the embodiment,” “the embodiments,” “one or more embodiments,” “some embodiments,” and “one embodiment” mean “one or more (but not all) embodiments of the present invention.”

[0088] Unless otherwise expressly stated, the terms “including,” “comprising,” “having,” and variations thereof mean “including but not limited to.”

[0089] Unless otherwise explicitly stated, the enumeration list of items does not imply that any or all items are mutually exclusive.

[0090] Unless otherwise expressly stated, the terms “a,” “an,” and “the” mean “one or more.”

[0091] Unless otherwise expressly stated, devices communicating with each other do not need to communicate continuously. Furthermore, devices communicating with each other may communicate directly or indirectly through one or more intermediaries.

[0092] The description of an embodiment having several components that communicate with each other does not imply that all such components are necessary. Rather, various optional components are described to illustrate various possible embodiments of the invention.

[0093] When a single device or item is described herein, it will be apparent that more than one device / item (whether or not they cooperate) may be used in place of a single device / item. Similarly, when more than one device or item (whether or not they cooperate) is described herein, it will be apparent that a single device / item may be used in place of more than one device or item, or that a different number of devices / items may be used in place of the number of devices or programs shown. The functionality and / or features of a device may alternatively be implemented by one or more other devices not explicitly described as having such functionality / features. Thus, other embodiments of the invention do not necessarily need to include the device itself.

[0094] For illustrative and descriptive purposes, the foregoing description of various embodiments of the invention has been presented. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in accordance with the foregoing teachings. The scope of the invention is intended to be limited not by this detailed description, but by the appended claims. The foregoing specification, examples, and data provide a complete description of the manufacture and use of the compositions of the invention. Because many embodiments of the invention can be made without departing from the spirit and scope of the invention, embodiments of the invention are present in the appended claims.

[0095] The foregoing description provides examples of embodiments of the invention, and variations and substitutions may be made in other embodiments.

Claims

1. A computer-implemented method, comprising the following operations: Multiple local devices were discovered; Receive user tags from each of the plurality of local devices; Store user tags for each of the plurality of local devices; A user receives a first position input from a global touchscreen of a global device for a mobile vehicle, wherein the first position input is a position relative to the display surface of the global touchscreen and indicates a stored user mark; Identify the local device among the plurality of local devices that is associated with the stored user tag; Determine the movement of the global device of the mobile vehicle that is different from the movement of the local device, wherein the local device has a motion sensor and wherein the local device is associated with a user; Motion information is retrieved from the motion sensors of the local device, wherein the motion information includes acceleration and indicates the movement of a user associated with the local device; The first position input is corrected to a second position input at the global touchscreen of the global device of the moving vehicle based on the motion information from the motion sensor of the local device; as well as Operations are performed at the global device of the moving vehicle based on the second location input.

2. The computer-implemented method according to claim 1, wherein, The local device is located away from the global device of the moving vehicle.

3. The computer-implemented method according to claim 1, further comprising: Register the plurality of local devices with the global device of the mobile vehicle.

4. The computer-implemented method according to claim 1, wherein, The user identifier includes a fingerprint.

5. The computer-implemented method according to claim 1, further comprising: The local device is notified to accept the second location input in order to perform processing based on the second location input.

6. The computer-implemented method according to claim 1, further comprising the following operations: Stores local device and user information.

7. A computer program product comprising program code executable by at least one processor of a global device of a moving vehicle to perform the following operations: Multiple local devices were discovered; Receive user tags from each of the plurality of local devices; Store user tags for each of the plurality of local devices; A first position input is received from a user touching the global touchscreen of the global device of the mobile vehicle, wherein the first position input is a position relative to the display surface of the global touchscreen and indicates a stored user mark; Identify the local device among the plurality of local devices that is associated with the stored user tag; Determine the movement of the global device of the mobile vehicle that is different from the movement of the local device, wherein the local device has a motion sensor and wherein the local device is associated with a user; Motion information is retrieved from the motion sensors of the local device, wherein the motion information includes acceleration and indicates the movement of a user associated with the local device; The first position input is corrected to a second position input at the global touchscreen of the global device of the moving vehicle based on the motion information from the motion sensor of the local device; as well as Operations are performed at the global device of the moving vehicle based on the second location input.

8. The computer program product according to claim 7, wherein, The local device is located away from the global device of the moving vehicle.

9. The computer program product of claim 7, wherein the program code is executable by at least one processor of the global device of the moving vehicle to perform the following operations: Register the plurality of local devices with the global device of the mobile vehicle.

10. The computer program product according to claim 7, wherein, The user identifier includes a fingerprint.

11. The computer program product according to claim 7, further comprising: The local device is notified to accept the second location input in order to perform processing based on the second location input.

12. The computer program product of claim 7, wherein the program code is executable by at least one processor of the global device of the moving vehicle to perform the following operations: Stores local device and user information.

13. A computer system for global equipment of a mobile vehicle, comprising: One or more processors, one or more computer-readable storage devices, and one or more computer-readable tangible storage devices; as well as Program instructions, stored on at least one of the one or more computer-readable tangible storage devices, for execution by at least one of the one or more processors via at least one of the one or more computer-readable storage devices to perform the following operations: Multiple local devices were discovered; Receive user tags from each of the plurality of local devices; User tags for each of the plurality of local devices are stored on the one or more computer-readable tangible storage devices; A user receives a first position input from a global touchscreen of a global device for a mobile vehicle, wherein the first position input is a position relative to the display surface of the global touchscreen and indicates a stored user mark; Identify the local device among the plurality of local devices that is associated with the stored user tag; Determine the movement of the global device of the mobile vehicle that is different from the movement of the local device, wherein the local device has a motion sensor and wherein the local device is associated with a user; Motion information is retrieved from the motion sensors of the local device, wherein the motion information includes acceleration and indicates the movement of a user associated with the local device; The first position input is corrected to a second position input at the global touchscreen of the global device of the moving vehicle based on the motion information from the motion sensor of the local device; as well as Operations are performed at the global device of the moving vehicle based on the second location input.

14. The computer system according to claim 13, wherein, The local device is located away from the global device of the moving vehicle.

15. The computer system of claim 13, wherein the operation further comprises: Register the plurality of local devices with the global device of the mobile vehicle.

16. The computer system according to claim 13, wherein, The user identifier includes a fingerprint.

17. The computer system of claim 13, wherein the operation further comprises: The local device is notified to accept the second location input in order to perform processing based on the second location input.

18. The computer system of claim 13, wherein the operation further comprises: Stores local device and user information.

Images