Error correction of virtual keyboard based on dynamic space model

Abstract

A method includes detecting a touch input performed by a user at a location on a virtual keyboard. The method further includes applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys based on a heat map of previous touch inputs performed by the user. The method further includes applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on comparing the location of the touch input to one or more predicted next keys. The method further includes selecting a specific key from the one or more first candidate keys and the one or more second candidate keys as indicated by the touch input.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 17 / 079,492, filed on October 25, 2020, entitled “VIRTUAL KEYBOARD ERRORCORRECTION BASED ON A DYNAMIC SPATIAL MODEL,” which is incorporated herein in its entirety. Background Art

[0003] With the popularity of mobile devices, the use of virtual keyboards has become popular. However, the use of virtual keyboards can be frustrating due to the difference between the user's expected position on the virtual keyboard and the actual tapping position. For example, a recent study on the types of typing errors showed that 37% of typing errors in English language virtual keyboards were due to missing space keys and 60% of typing errors were related to the space key, such as confusion between the space key and the C, V, B, and N letters in the last row of the virtual keyboard. This may be due to the user having wide fingers, tapping with only one hand, not using sufficiently precise touch, touch screens, and the inherent inaccuracy of gesture-based keystroke entry, etc.

[0004] The description of the background technology provided herein is intended to present the context of the present disclosure in general. To the extent described in this background technology section, the work of the presently named inventors, and aspects of the description that are not otherwise prior art at the time of filing, are neither explicitly nor implicitly admitted as prior art with respect to the present disclosure. Summary of the invention

[0005] A method may include detecting a touch input performed by a user at a location on a virtual keyboard, applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys based on a heat map of previous touch inputs, applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on a comparison of the location of the touch input with one or more predicted next keys, and selecting a specific key from the one or more first and second candidate keys as indicated by the touch input.

[0006] In some embodiments, the spatial model determines a global key center offset based on a heat map of previous touch inputs near the key center using a first Gaussian distribution, and the spatial model maps the touch input based on the global key center offset. In some embodiments, the method further includes applying a key-level center offset to the touch input based on a second Gaussian distribution, wherein the spatial model maps the touch input based on the key-level center offset. In some embodiments, the spatial model provides one or more of the following: a global level correction in which all keys in the keyboard layout share a parameter, a cluster-level correction in which all keys in the same cluster share a parameter, or a key-level correction in which each key has its own Gaussian parameter. In some embodiments, the method further includes adjusting the visual appearance of at least a portion of the virtual keyboard based on the heat map. In some embodiments, adjusting the visual appearance of at least a portion of the virtual keyboard includes adjusting the visual appearance of the space bar to at least expand the space above or below the space bar. In some embodiments, the method further includes determining one or more words including characters corresponding to the particular key, and modifying the one or more words based on the language model to add or remove spaces. In some embodiments, the method further includes updating the heat map to discard previous touches that are older than a predetermined amount of time. In some embodiments, the method further includes determining an identity of a user providing the touch input, wherein the identity is determined based on authentication input provided by the user, and retrieving a heat map corresponding to the user.

[0007] A computing device may include a processor and a memory having instructions stored thereon, which when executed by the processor cause the processor to perform operations, including: detecting a touch input performed by a user at a location on a virtual keyboard, applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys based on a heat map of previous touch inputs, applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on a comparison of the location of the touch input with one or more predicted next keys, and selecting a specific key from the one or more first and second candidate keys as indicated by the touch input.

[0008] In some embodiments, the spatial model determines a global key center offset based on a heat map of previous touch inputs near the key center using a first Gaussian distribution, and the spatial model maps the touch input based on the global key center offset. In some embodiments, the operation further includes applying a key-level center offset to the touch input based on a second Gaussian distribution, wherein the spatial model further maps the touch input based on the key-level center offset. In some embodiments, the spatial model provides one or more of the following: a global-level correction in which all keys in a keyboard layout share one parameter, a cluster-level correction in which all keys in the same cluster share one parameter, or a key-level correction in which each key has its own Gaussian parameter. In some embodiments, the operation further includes adjusting the visual appearance of at least a portion of the virtual keyboard based on the heat map. In some embodiments, adjusting the visual appearance of at least a portion of the virtual keyboard includes adjusting the visual appearance of the space bar to at least expand the space above or below the space bar.

[0009] A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more computers, causes the one or more computers to perform operations, the operations comprising: detecting a touch input performed by a user at a location on a virtual keyboard, applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys based on a heat map of previous touch inputs, applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on a comparison of the location of the touch input with one or more predicted next keys, and selecting a specific key from the one or more first and second candidate keys as indicated by the touch input.

[0010] In some embodiments, the spatial model determines a global key center offset based on a heat map of previous touch inputs proximate to the key center using a first Gaussian distribution, and the spatial model maps the touch input based on the global key center offset. In some embodiments, the operation further includes applying a key-level center offset to the touch input based on a second Gaussian distribution, wherein the spatial model further maps the touch input based on the key-level center offset. In some embodiments, the operation further includes adjusting the visual appearance of at least a portion of the virtual keyboard based on the heat map. In some embodiments, adjusting the visual appearance of at least a portion of the virtual keyboard includes adjusting the visual appearance of the space bar to expand the space at least above or below the space bar. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0012] Figure 1 is a block diagram of an example network environment according to some embodiments described herein.

[0013] Figure 2 is a block diagram of an example computing device according to some embodiments described herein.

[0014] Figure 3A is an example illustration of a heat map generated by touch input on a virtual keyboard according to some embodiments described herein.

[0015] Figure 3B According to some embodiments described herein, Figure 3A Example illustration of the key center offset for each key as determined by the heat map shown in .

[0016] Figure 4 is a flow chart of an example method for applying key corrections to keys of a virtual keyboard according to some embodiments described herein.

[0017] Figure 5A is an example illustration of the x- and y-dimensional differences between the center of a key on a virtual keyboard and a touch input from a user.

[0018] Figure 5B is an example illustration of touch input for the space bar on the virtual keyboard.

[0019] Figure 5C is an example diagram of calculation of x-dimensional change and y-dimensional change of touch input with reference to the space bar according to some embodiments described herein.

[0020] Figure 5D is an illustration of another example of a Gaussian distribution in which a spatial model is used to increase the number of touch inputs mapped to a space bar according to some embodiments described herein.

[0021] Fig. 6A is an example illustration of two different ways to modify the position of the space bar according to some embodiments described herein.

[0022] Figure 6B is an example illustration of a virtual keyboard in which a heat map indicates that a touch input from a user to a space bar is at an angle.

[0023] Figure 6C is an example illustration of a virtual keyboard with visual distinctions between letter keys and function keys according to some embodiments described herein.

[0024] Fig.6D is an example illustration of a virtual keyboard with a spacebar key shifted downward and offset from adjacent keys according to some embodiments described herein.

[0025] Fig. 6E is an example illustration of a virtual keyboard with an empty space in place of a spacebar key in accordance with some embodiments described herein.

[0026] Fig. 6F is an example illustration of the use of isolation above and below the spacebar key according to some embodiments described herein.

[0027] Figure 7 is a flow chart of an example method for selecting a specific key based on touch input received on a virtual keyboard according to some embodiments described herein. DETAILED DESCRIPTION

[0028] In some embodiments, a keyboard application (e.g., which may be part of a device operating system or a separate application) detects touch input by a user at a location on a virtual keyboard. For example, after having entered "mor", the user touches a point on the virtual keyboard near the "b", "n", and space keys. The keyboard application applies a dynamic spatial model to map the touch input to one or more first candidate keys of the virtual keyboard. In this example, the spatial model maps the touch input to one or more first candidate keys (e.g., "b", "n", and space keys) based on a heat map of previous touch inputs performed by the user. The keyboard application applies a language model to map the touch input to one or more second candidate keys of the virtual keyboard. For example, according to the language model, there is a 50% probability that the letter being entered is "n" because it corresponds to a word such as "morning" that may occur commonly; there is a 20% probability that the letter being entered is "a" because it corresponds to "moral" or "moratorium" that may occur less frequently; there is a 10% probability that the letter being entered is "b" because it corresponds to "morbid" or even less common; and there is a 0% probability that the letter is a space (corresponding to the selection of the space bar key) because "mor" is not a correct word (in the English language, and in the user's proprietary dictionary, if such a dictionary is available). The keyboard application selects a particular key from one or more first and second candidate keys as indicated by the touch input. For example, if the spatial model indicates that there is a 70% probability that the touch input corresponds to the "n" key based on the touch location and the heat map, and the language model indicates that there is a 50% probability that the touch input corresponds to the "n" key based on word prediction, the keyboard application selects "n" based on the combined result. Using the combined result can provide the user with reduced typing errors and allow faster data entry. Thus, an improved human-computer interface is provided.

[0029] In some embodiments, the keyboard application modifies a portion of the virtual keyboard based on the heat map. For example, if the keyboard application determines that the previous input and the touch input are above the default key center, the keyboard application may move the display of a portion of the virtual keyboard (e.g., the space bar and / or other keys) to a higher position. In another example, the keyboard application may not move the display, but instead interpret a touch received a predetermined number of pixels above the space bar as corresponding to the space bar.

[0030] Network environment 100

[0031] Figure 1A block diagram of an example network environment 100 is illustrated. In some embodiments, the network environment 110 may include one or more mobile devices, such as mobile devices 115a, 115n. The mobile device 115a may include a keyboard application 103. The keyboard application 103 may receive touch input from a user and select specific keys for display on a virtual keyboard based on a spatial model and a language model.

[0032] Mobile device 115 can communicate with a network such as network 105. Network 105 can be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switch or hub connection, etc. In some embodiments, network 105 can include communication between devices, such as using a peer-to-peer wireless protocol (e.g., An example of peer-to-peer communication between two mobile devices 115a and 115b is shown by arrow 132.

[0033] The network environment 100 may also include one or more server systems, such as Figure 1 The server system 101 in the example of . The server system 101 may include a database 199 or other storage device. In some embodiments, the server system 101 may send updates for the keyboard application 103 to the mobile device 115.

[0034] For ease of explanation, Figure 1 One box is shown for server system 101 and database 199, and two boxes are shown for mobile devices 115a and 115n. Server boxes 101 and 199 can represent multiple systems, server devices, and network databases, and boxes can be provided in different configurations than shown. For example, server system 101 can represent multiple server systems that can communicate with other server systems via network 105. In some embodiments, for example, server system 101 can include a cloud hosting server. In some examples, database 199 and / or other storage devices can be provided in a server system box separate from server system 101, and can communicate with server system 101 and other server systems via network 105.

[0035] There may be any number of mobile devices 115. Each mobile device 115 may be any type of electronic device, such as a portable device using a virtual keyboard, a cellular phone, a smart phone, a tablet computer, a camera, a smart display, a wearable device (e.g., display glasses or goggles, a watch, a headset, an armband, jewelry, etc.), a personal digital assistant (PDA), a media player, a gaming device, etc. The mobile device 115 may also include a local database similar to the database 199 or other storage. In some embodiments, the network environment 100 may not have all of the components shown and / or may have other components, including other types of components in addition to or instead of those described herein.

[0036] In various embodiments, users 125 may communicate with server system 101 and / or with each other using respective mobile devices 115a, 115n. In some examples, users 125 may interact with each other via keyboard applications 103 running on respective mobile devices 115 and / or interact with server system 101 via network services implemented on server system 101, such as messaging applications, social networking services, image hosting services, or other types of network services. For example, respective mobile devices 115a, 115n may communicate data to and from one or more server systems, such as server system 101.

[0037] In some embodiments, server system 101 may provide appropriate data, such as keyboard application 103 updates, to mobile devices 115a, 115n so that each mobile device 115 may receive communicated or shared content uploaded to server system 101 and / or network services.

[0038] Example computing device 200

[0039] Figure 2 is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. In one example, computing device 200 can be mobile device 115 used to implement keyboard application 103. Computing device 200 can be any suitable computer system, server, or other electronic or hardware device as described above.

[0040] One or more of the methods described herein may be run as a standalone program that can be executed on any type of computing device, a program that runs on a web browser, a mobile application ("app") that runs on a mobile computing device (e.g., a cellular phone, a smart phone, a smart display, a tablet computer, a wearable device (a watch, an armband, jewelry, a head-mounted device, virtual reality goggles or glasses, augmented reality goggles or glasses, a head-mounted display, etc.), a laptop computer, etc.). In the primary example, all computations are performed within the mobile application (and / or other applications) on the mobile computing device. However, a client / server architecture may also be used, such as where the mobile computing device sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, computations may be divided between the mobile computing device and one or more server devices.

[0041] In some embodiments, computing device 200 includes processor 202, memory 204, input / output (I / O) interface 206, display device 208, and database 210. Processor 202, memory 204, I / O interface 206, display device 208, and database 210 may be coupled via bus 220.

[0042] Processor 202 may be one or more processors and / or processing circuits used to execute program code and control the basic operations of computing device 200. "Processor" includes any appropriate hardware system, mechanism or component for processing data, signals or other information. Processors may include systems with a general-purpose central processing unit (CPU) with one or more cores (e.g., single-core, dual-core or multi-core configuration), multiple processing units (e.g., microprocessor configuration), graphics processing units (GPUs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), complex programmable logic devices (CPLDs), dedicated circuits for implementing functions, dedicated processors for implementing processing based on neural network models, neural circuits, processors optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, processor 202 may include one or more coprocessors that implement neural network processing. In some embodiments, processor 202 may be a processor that processes data to generate a probabilistic output, for example, the output generated by processor 202 may be inaccurate or may be accurate within a range from the expected output. Processing does not need to be limited to a specific geographic location or have time constraints. For example, a processor may perform its functions in real time, offline, or in batch mode, etc. Portions of a process may be performed at different times and in different locations by different (or the same) processing systems.A computer may be any processor in communication with a memory.

[0043] The memory 204 is generally provided in the computing device 200 for access by the processor 202, and may be any suitable processor-readable storage medium suitable for storing instructions for execution by a processor or a set of processors, such as a random access memory (RAM), a read-only memory (ROM), an electrically erasable read-only memory (EEPROM), flash memory, etc., and is located separately from and / or integrated with the processor 202. The memory 204 may store software operated by the processor 202 on the computing device 200, including the keyboard application 103.

[0044] The I / O interface 206 can provide functionality to enable the computing device 200 to dock with other systems and devices. The docking device can be included as part of the computing device 200, or can be separate and communicate with the computing device 200. For example, a network communication device, a storage device (e.g., a memory and / or database 199), and an input / output device can communicate via the I / O interface 206. In some embodiments, the I / O interface can be connected to an interface device, such as an input device (keyboard, pointing device, touch screen, microphone, camera, scanner, sensor, etc.) and / or an output device (display device, speaker device, printer, motor, etc.). For example, when a user provides touch input, the I / O interface 206 transmits data to the keyboard application 103.

[0045] Some examples of docking devices that can be connected to I / O interface 206 can include one or more display devices 208, which can be used to display content, such as images, videos and / or user interfaces of output applications as described herein, and receive touch (or gesture) input from the user. For example, display device 208 can be used for virtual keyboard on display screen. Display device 208 can include touch detection capability (for example, via resistance, capacitance or other circuits that can detect touch). For example, display device 208 can detect when the user touches or makes their fingers or input pens approach a specific area of ​​display device 208, such as individual pixels of display device, pixel clusters of display device, etc. Display device 208 can be connected to computing device 200 via local connection (for example, display bus) and / or via networking connection, and can be any appropriate display device. Display device 208 can include any appropriate display device, such as liquid crystal display (LCD), light emitting diode (LED) or plasma display screen, cathode ray tube (CRT), television, monitor, touch screen, three-dimensional display screen, or other visual display device. For example, display device 208 may be a flat display screen provided on a mobile device, multiple display screens embedded in an eyeglass form factor or head mounted device, or a monitor screen for a computing device.

[0046] In some embodiments, for example, when touch capabilities are integrated into the display device 208, such as a touch screen display, the display device 208 can directly act as an input device. In some embodiments, a touch recognition layer can be provided in the computing device 200 that is separate from the display device 208. In these embodiments, the touch recognition layer can be part of the I / O interface 206 and can detect touch locations, for example, while the display device 208 displays a virtual keyboard or other content. For the purposes of this document, it is assumed that the computing device 200 includes hardware touch detection capabilities, whereby touch inputs provided by the user are detected and mapped to specific locations of the virtual keyboard displayed on the display device 208.

[0047] Database 210 stores data related to keyboard application 103. For example, database 210 may store a heat map of previous touches from a user, a personal dictionary specific to the user, etc. In some embodiments, the heat map is updated to discard previous touches that are older than a predetermined amount of time (e.g., last 24 hours, last week, last hour, etc.).

[0048] Keyboard Application 103

[0049] The keyboard application 103 may include a spatial module 212 , a language module 214 , and a user interface module 216 .

[0050] The spatial module 212 applies a dynamic spatial model to map the touch input to one or more candidate keys of the virtual keyboard. In some embodiments, the spatial module 212 includes an instruction set that can be executed by the processor 202 to map the touch input to one or more candidate keys of the virtual keyboard. In some embodiments, the spatial module 212 is stored in the memory 204 of the computing device 200 and can be accessed and executed by the processor 202.

[0051] In some embodiments, the spatial module 212 receives touch input from the user via the I / O interface 206 and / or the display device 208. The spatial module 212 can generate a heat map from the touch input. In some embodiments, the spatial module 212 generates a heat map based on a fixed period of time, such as the last 24 hours, the last week, etc. Maintaining the most recent heat map can advantageously take into account changes in the user's touch input pattern, such as when the user changes the housing or cover of the mobile device, grips the phone in a different way, holds the phone in a different way (e.g., in portrait or landscape orientation), has a hand injury, or otherwise changes the input mode, etc. The spatial module 212 can store the heat map in the database 210. In some embodiments, when there is not enough data to generate a heat map of previous touch inputs performed by the user, the spatial module 212 can use a default heat map as a starting point.

[0052] In some embodiments, multiple users may use the same mobile device 115. The spatial module 212 can generate multiple heat maps for the mobile device 115 and retrieve heat maps for users based on the heat maps corresponding to specific users. The spatial module 212 can determine the identity of the user providing the touch input based on the authentication input provided by the user, such as biometric data (e.g., fingerprint, camera identification, etc.), password, pass pattern, etc. The authentication input is only used for authentication purposes if specifically enabled by the user, and is not used for touch input purposes. The user is provided with the option to enable or disable specific types of authentication, for example, the user can completely turn off biometric authentication. Authentication is performed locally on the device, and no user-identifiable data is transmitted to any other device.

[0053] In some embodiments, the spatial module 212 maps the touch input to one or more first candidate keys based on a heat map of previous touch inputs made by the user. For example, if the heat map indicates that the user always taps at the center of each key, the spatial module 212 can map the touch input to the center of the candidate key (i.e., the default key center). In another example, if the heat map indicates that the user always taps at the upper left of the key on the virtual keyboard, the spatial module 212 can map the touch input to correspond to the key at the lower right of the touch input. The spatial module 212 can retrieve the heat map for the user based on being identified as a user accessing the computing device 200. The heat map can indicate different offsets from the tapping position for different keys, for example, different offsets from the key center for the left and right portions of the virtual keyboard (which may be accessed by different fingers of different hands), different offsets for different rows (e.g., which may be at different distances from the user's fingers as the user grasps the device and types on the virtual keyboard), or different offsets for individual keys. In some embodiments, the spatial module 212 performs adjustments based on the heat map as described below.

[0054] The spatial module 212 can be used to identify the location of the actual key center specific to the user and how it differs from the location of the default key center in order to determine the difference between the locations. Figure 3AIt is an example illustration 300 of a heat map generated by previous touch inputs on a virtual keyboard obtained from a volunteer user during a test data entry session. In this example, the touch input data is obtained from 50 volunteer users, and they each entered 50 sentences. Illustration 300 includes the distribution of touch inputs of all 50 users on the virtual keyboard for 2500 sentences (about 12000 words). Gray points represent actual touch inputs of users, red points represent default key centers (i.e., key centers from keyboard layouts), blue points represent key centers calculated from user touch inputs, and green points represent touch inputs that have triggered key corrections (users have corrected because virtual keyboards have interpreted wrong keys). As can be seen, green points appear near key edges. Figure 3A It was demonstrated that volunteer users in this dataset tended to type lower than the default key center, especially for keys on the first row of the virtual keyboard.

[0055] Although Figure 3A A particular type of heat map is shown that is generated from aggregated data obtained from multiple volunteer users, but it will be appreciated that different users may have different touch patterns and corresponding offset points at which they touch different keys. In actual use, as further described below, the spatial module 212 can generate a heat map locally on the device for each user as the user uses the on-screen virtual keyboard, and perform corrective actions when necessary. Over time, as the user enters text using the virtual keyboard, the heat map represents the pattern of the user's touch input, such as the user touching the "a" key closer to its boundary with the "s" key, the user touching the space bar closer to its boundary with the "c", "v", "b", and "n" keys toward the top, and so on.

[0056] Figure 3B is an example diagram 350 of a key center offset measured as a function of the key center offset according to some embodiments described herein. Figure 3A . The distance (in pixels) between the touch input and the geometric key center of each key is determined based on the heat map shown in . In this example, the offset on the x-axis (x_offset) is illustrated in blue, while the offset on the y-axis (y_offset) is illustrated in red. The leftmost data point is the offset of the space bar. Next are the keys for the English alphabet. The diagram shows that some letters have more significant offsets than others. For example, the letter "j" has the highest x_offset and y_offset.

[0057] In some embodiments, the spatial module 212 generates a dynamic spatial model by determining the key offset between the default key center of the virtual keyboard and the actual key center as indicated by the user's touch input, and dynamically modifying based on the key offset. In some embodiments, the spatial model uses a Gaussian distribution. In some embodiments, the spatial module 212 maintains different spatial models for different keyboard layouts.

[0058] The following definitions are used in the following equations: x, y are the coordinates of the touch input and the unit is pixels (pixel coordinates); key is the key that the user intends to hit; key actual is the actual key center position of the key being struck from the virtual keyboard; d(x, y, key default ) is the difference between the user's actual touch input and the default key center of the key, which is a vector; d(x, y, key actual ) is the difference between the user's actual touch input and the actual key center; d(key actual ,key default ) is the key center offset and is defined as the difference between the user's actual key center and the default key center; σ d is the standard of the current Gaussian distribution of the spatial model; and E[d()] is the expectation of a specific distance.

[0059] In one embodiment, the spatial module 212 determines the alignment score using the following equation:

[0060] Equation 1AlignScore(x, y, key) = d 2 (x, y, key default ) / σ d 2

[0061] The difference between the touch input and the default button center can be considered as the sum of the touch input to the actual button center and the actual button center to the default button center, as defined by the following equation:

[0062] Equation.2d(x, y, key default ) = d(x, y, key actual )+d(key actual ,key default )

[0063] The distribution of touch input for a specific key around the actual key center is defined as follows:

[0064] Equation 3E[d(x, y, key actual )]=0

[0065] thereby:

[0066] Equation 4E[d(x, y, key default )] = E[d(x, y, key actual )]+E[d(key actual ,key default )]=d(key actual ,key default )

[0067] to produce the following equation:

[0068] Equation 5 keyoffset = d(key actual ,key default )=E[d(x,y,key default )]

[0069] The above equation for the key offset reveals that the spatial module 212 predicts the key center offset based on the default key center and the touch input from the user. In some embodiments, this is an unsupervised algorithm without complex learning and calculation, which means that machine learning is not part of the spatial module 212. In some embodiments, the spatial module 212 uses weights to add the key center offset to the default key center to obtain a personalized key center:

[0070] Equation .6keycenter personalized =key default +weight*d(key actual ,key default )

[0071] The spatial module 212 can determine a key offset for each key (including the space bar). Key offsets may be caused by a user's finger error, such as when the finger is wider than the touch area; the user's position / angle of view when typing; and the user's typing habits, such as left hand, right hand, single hand or double hand. In some embodiments, the spatial module 212 maps the touch input to a candidate key of the virtual keyboard based on the key offset. For example, in the case where the touch input is closer to "e" than "d", the spatial module 212 can determine that the candidate key is "d" based on the actual key center position being higher than the default key center. In some embodiments, the spatial module 212 adjusts the position of at least a portion of the virtual keyboard based on the spatial model. For example, continuing with the previous example, the spatial module 212 can instruct the user interface module 216 to adjust the position of the "d" key to a higher position on the virtual keyboard (e.g., by interpreting a touch of a predetermined number of pixels above the "d" key as indicating the "d" key rather than the actual key that may be touched above it, such as "e" or "r") to reduce key offset.

[0072] Typing errors due to the position and visual angle of the virtual keyboard typically remain constant while typing a word. As a result, the spatial module 212 applies word-level learning by using the key center offset as a constant for word prediction because the key center offset will be less likely to change between letters. The spatial module 212 determines the key offset for the first part of the word and then determines the second part of the word. For example, if the user is typing "internationalization" and has already typed "internation" when calculating the spatial score for the letter "a", the spatial module 212 determines the key center offset based on the touch input of "internation", applies the key center offset to the key center of the spatial model; and calculates a personalized spatial score.

[0073] In some embodiments, the spatial module 212 uses one or more of the following factors in the spatial model: a key center sampling size, which specifies the minimum number of touch inputs to be learned; a tap key center offset weight, which is the weight of the key center offset applied to the key center for tap typing; and a gesture key center offset weight, which is the weight of the key center offset applied to the key center for gesture typing.

[0074] In some embodiments, rather than performing word-level learning, the spatial module 212 performs in-memory learning. This addresses deficiencies in word-level learning in cases where word lengths are too short and in instances where noise affects the personalized key centers enough to cause inaccuracies. The spatial module 212 performs in-memory learning by saving user history (with specific permission from the user for such secure in-memory storage) in a cache stored in the database 210. More specifically, when the user begins typing a word, the spatial module 212 retrieves the key center offset from the in-memory history, applies the key center offset to the key center, calculates the spatial score, and updates the in-memory history. In some embodiments, the spatial module 212 uses the following factors in the spatial model: a key center sample size, which specifies the minimum number of touch inputs to be learned; and a tap key center offset weight, which is the weight of the key center offset applied to the key center of a tap-typed key.

[0075] In some embodiments, the spatial module 212 performs key correction for the space bar of the virtual keyboard by changing the interpretation of the touch input to the space bar. For example, even when the user provides a touch input that is not directly on the space bar, the spatial module 212 can interpret it as corresponding to a touch of the space bar. The spatial module 212 can use a Gaussian distribution to determine how to perform key correction.

[0076] In some embodiments, the spatial module 212 determines the distribution of touch inputs around the key centers of the spatial model using a Gaussian distribution. For example, the spatial module 212 may use a Gaussian model param function with the following parameters: x-axis offset, y-axis offset, x-axis variance, y-axis variance (where, if this is missing, the x-axis variance is used), where the default is to treat the distribution as the covariance of the two dimensions of the distribution as independent distributions, and the total number of data points. Based on the definition of the Gaussian parameters, the spatial module 212 maintains the following different levels of personalized parameters: global-level correction based on global key center offset, where all keys in the keyboard layout share one parameter; cluster-level correction of cluster key center offset, where all keys in the same cluster share one parameter; and key-level correction of key-level center offset, where each key has its own Gaussian parameter.

[0077] In some embodiments, in addition to the spatial model for global key center offset, the spatial module 212 also generates a separate spatial model for key correction. In this way, the spatial module 212 is responsible for predictions that occur before a space is entered and after a space is entered. For example, the spatial module 212 uses the first spatial module to make word predictions before the end of a word, for example, "mornin" has a key prediction of "g", and even if the user types "h" (which is immediately to the right of "g" on the virtual keyboard), the spatial module 212 will select the letter "g". The spatial module 212 uses the second spatial module to make word predictions when the entire word has been written, for example, the spatial module 212 converts the written word "jappy" into "happy".

[0078] The spatial module 212 can maintain different spatial models for the same user. For example, different spatial models can be associated with different device orientations such as portrait or landscape. Further, if the device is used by multiple users, different on-device spatial models can be maintained for different users with the permission of the corresponding users. The generation, storage and / or use of user-specific heat maps, spatial models, and / or the determination of candidate keys based on touch input are all performed with the permission of the specific user. Each user is provided with the option to enable or disable such personalization. For example, the user can be presented with a menu that clearly indicates the type of information that can be adopted (e.g., previous touch history) and the benefits of personalization (e.g., reduced typing errors and faster data entry). The user can be provided with the option to enable / disable personalization at any time. User data such as previous touch inputs and derived information such as heat maps or spatial models are stored locally on the device and are specifically used for the purpose of improving data entry via the virtual keyboard. Such information is discarded when the user disables personalization and / or chooses to delete the data. If the user disables the personalization feature, no information about the user's data entry using the virtual keyboard can be accessed or stored. The space module 212 operates entirely on-device.

[0079] Figure 4 4 is a flow chart of an example method 400 for applying key correction to each key in a virtual keyboard according to some embodiments described herein. In this example, the spatial module 212 receives 405 touch input data from a user, retrieves 410 the user touch history from the database 210 to apply a global level key center offset to the touch input data, determines 415 whether a key level correction (KC) should be applied to the touch input data to correct for a particular key, and updates 420 a spatial model for a global key center offset.

[0080] For this embodiment, the spatial module 212 does not apply key-level correction when certain prerequisites are met. For example, the spatial module 212 does not apply key-level correction when the sample size (e.g., the words typed) is below a predetermined threshold, the spatial module 212 applies the learned weights of the key center offset to the touch input, or the spatial module 212 applies the learned weights of the key center variance to the touch input. In some embodiments, although the space bar is a key, the spatial module 212 applies different rules to it because the space bar has a larger x-range. For example, for touch input corresponding to the space bar, because the rules are different from those for letter keys, the spatial module 212 may not update the spatial model for global key offsets. Method 400 can be repeated as additional touch input data is received.

[0081] In some embodiments, the spatial module 212 uses a Gaussian distribution as part of the key-level correction to generate the spatial scores. Figure 5A 5 is a diagram 500 of how the spatial module 212 determines the x-dimension difference (dx) and y-dimension difference (dy) between the center of a key on a virtual keyboard and a touch input from a user, according to some embodiments. For keys where the width or height of a key (e.g., the space bar) is larger than a normal key (e.g., "n"), the user may not touch at the center of the key. For example, Figure 5B is an illustration 525 of a touch input for the space bar, which shows that in addition to not touching in the center of the space bar, some users even touch outside the space bar.

[0082] In some embodiments, the spatial module 212 uses the following equation to determine the bounds of a key:

[0083] Equation.7x_lower_bound=center+x-key_width / 2+common_key_width / 2

[0084] Equation.8x_upper_bound=center_x+key_width / 2-common_key_width / 2

[0085] Equation.9y_lower_bound=center_y-key_height / 2+common_key_height / 2

[0086] Equation.10y_upper_bound=center_y+key_height / 2-common_key_height / 2

[0087] Figure 5C 550 is an example diagram of how the x-dimension change (dx) and the y-dimension change (dy) are calculated for touch input with reference to the space bar. Figure 5C Illustrated the default key centers of the C key, N key, and space bar as gray circles. Figure 5C The touch input icon is a hollow circle. Figure 5C As shown in , the x_lower_bound of the space bar is the same value as the center x of key "C", and the x_upper_bound of the space bar is the same value as the center x of key "N".

[0088] For keys whose width is greater than that of normal keys (e.g., for the space bar), the spatial module 212 determines dx using the following equation:

[0089] Equation.11 If touch_x > x_lower_bound & touch_x < x_upper_bound, then dx = 0

[0090] Equation.12 If touch_x < x_lower_boun, then dx = x_lower_bound - touch_x

[0091] Equation.13 If touch_x > x_upper_bound, then dx = touch_x - x_upper_bound

[0092] For keys whose height is greater than that of a normal key, the spatial module 212 uses equations similar to Equations 11 - 13 but with y replacing x to determine dy.

[0093] In some embodiments, the spatial module 212 generates a spatial score using an extended version of the Gaussian distribution as part of key - level calibration. Figure 5D FIG. 575 is a diagram of a Gaussian distribution used by the spatial model that has additional overlap between the C and N keys and the space bar on the virtual keyboard. In this example, the spatial module 212 uses a gaussian_target_size_factor with a value of 1.5 as illustrated by the dashed line in Figure 5D . The spatial module 212 multiplies the width and height of all keys by the gaussian_target_size_factor, and also multiplies the normal key width and normal key height by the gaussian_target_size_factor. As a result, a touch input between the "C" key and the space bar or between the "N" key and the space bar can map the touch input to the candidate keys "C" and the space bar or "N" and the space bar, rather than using a less extended Gaussian distribution with less overlap, so that the spatial module 212 will map the touch input to the candidate keys "C" or the space bar or the candidate keys "N" or the space bar.

[0094] Continue Figure 5D , the spatial module 212 obtains the error x_lower_bound and x_upper_bound of the space bar by multiplying the width and height of all keys by the gaussian_target_size_factor. Since the x_lower_bound of the space bar is less than the center of the C key and the x_upper_bound of the space bar is greater than the center x of the N key, compared with using a Gaussian distribution without the gaussian_target_size_factor, dx causes the spatial module 212 to associate a larger spatial score to the space bar.

[0095] In some embodiments, the spatial module 212 performs key correction for the keys of the virtual keyboard by adjusting the visual appearance of at least a portion of the virtual keyboard. Adjusting the visual appearance of the virtual keyboard can allow greater clarity about the key boundaries of the virtual keyboard as mapped by the spatial module 212. For example, the spatial module 212 moves the position of the entire virtual keyboard to reduce key offsets, such as by making the virtual keyboard display closer to the key positions as indicated by the heat map. In another example, the spatial module 212 modifies the visual appearance of specific keys based on the heat map. For example, the spatial module 212 can determine that the key offset of the key "N" is the largest of all letter keys based on the heat map, and as a result, increase the demarcation size of the key "N" on the virtual keyboard (touches within which can be mapped to the "N" key by the spatial module 212).

[0096] In some embodiments, the spatial module 212 adjusts the visual appearance of the space bar. For example, in some embodiments, where the spatial module 212 determines based on the heat map that the touch input should correspond to the space bar, but the touch input is outside the space bar, the spatial module 212 commands the user interface module 216 to generate a user interface in which the visual appearance of the space bar is adjusted based on the heat map of the touch input. The module 212 can command the user interface module 216 to modify the display by raising the keyboard position (e.g., raising it by a few pixels, such as about half the height of the space bar), extending the space bar clickable area until it reaches the system navigation bar (if in use) or other UI elements, extending the space bar clickable area below the space bar (e.g., extending about half the height of the space bar), or raising the keyboard position to a height determined based on the history of touch input from the user. In some embodiments, the spatial module 212 can command the user interface module 216 to generate a user interface in which the visual appearance of the space bar is adjusted by changing the demarcation of the space bar.

[0097] Fig. 6A 600 is an example diagram of two different ways to modify the position of the space bar according to embodiments described herein. In this example, touch input received in the isolation above the space bar and the clickable space from below the space bar to the system navigation bar 601 is still considered to correspond to the selection of the space bar. In some embodiments, the space module 212 modifies the visual appearance of the space bar with more pixels or modifies the mapping of touch input to correspond to the space bar.

[0098] Figure 6B605 is an example illustration of a virtual keyboard, wherein a heat map indicates that the touch input for the space bar from the user is at an angle. For example, if the user tends to touch at the top of the space bar near the left side of the "C" key, and gradually lower towards the right side, the lowest point is just above the space bar near the "N" key. In this example, the heat map of the space bar key is represented by an angled dotted line 606. For example, this will occur when the user mainly types with their right hand and the thumb provides the touch input at an angle higher than other fingers. In some embodiments, the spatial module 212 uses a heat map including the touch input at a higher angle to detect the touch input in the overlapping area, and uses a spatial model to explain whether the touch input is intended for C, V, B or N keys or the space bar. As a result, although the visual appearance of the space bar remains unchanged, the spatial module 212 uses more pixels to map the touch input so as to correspond to the space bar at an angle, so that the touch closer to the pixels of the C, V, B or N keys on the screen or even overlapping with those keys is assigned to the space bar.

[0099] Figure 6C 625 is an example illustration of a virtual keyboard with visual distinctions between letter keys and function keys according to some embodiments described herein. In this example, the function keys are within a red rectangle (which is not part of the virtual keyboard and is included here for illustration purposes). Letters are displayed in white, and other keys are displayed with visual distinctions. For example, the Caps Lock key, number / character function keys, comma key, period key, and backspace key are displayed in gray, the Enter button is displayed in blue, and the Search button and Space key are displayed in white but as part of a row of keys that include keys of other colors.

[0100] Fig.6D is an example illustration 630 of a virtual keyboard in accordance with some embodiments described herein in which the user interface module 216 modifies the visual appearance of the space bar by shifting the space bar key downward, thereby enabling a distinction between touch input mapped to the space bar and the C, V, B, and N keys.

[0101] Fig. 6E 650 is an example of a less spatial design that enables the distinction of touch inputs mapped between the space bar and the C, V, B, or N keys according to some embodiments described herein. In this example, the user interface module 216 generates a virtual keyboard with no visual outline for the space bar. In addition, the spatial module 212 determines that touch inputs that are not directly on the C, V, B, or N keys correspond to the space bar. As a result, a greater number of touch inputs are mapped to the space bar unless they are directly on one of the C, V, B, or N keys.

[0102] Fig. 6F 670 is an example illustration of using isolation above and below the space key to resolve confusion between the space key and the C, V, B, or N keys according to some embodiments described herein. In this use, the user interface module 216 generates a virtual keyboard in which the space key has no visual outline and there are isolations above and below the space key, so that an even greater number of touch inputs that are not directly on the C, V, B, or N keys correspond to the space key.

[0103] The language module 214 applies the language model to map the touch input to one or more candidate keys of the virtual keyboard. In some embodiments, the language module 214 includes an instruction set that can be executed by the processor 202 to map the touch input to one or more candidate keys of the virtual keyboard. In some embodiments, the language module 216 is stored in the memory 204 of the computing device 200 and can be accessed and executed by the processor 202.

[0104] In some embodiments, the language module 214 maps the touch input to one or more second candidate keys of the virtual keyboard, which may be different from the one or more first candidate keys to which the spatial module 212 maps the touch input. For example, the spatial module 212 may determine based on the spatial model that the fourth letter is "s" based on the proximity of the touch input to the letter "s" when the first three letters are "rat", but the language module 214 maps the touch input to the letter "e" because the language model indicates that "rate" is a more common word than "rats" or the user types "rate" more frequently than "rats" (if the user permits the user typing data to be used for such determination). In some embodiments, the language module 214 may be different for different languages.

[0105] The language module 214 can map the touch input to one or more second candidate keys based on comparing the location of the touch input to one or more predicted next keys. In some embodiments, the language module 214 applies a Gaussian distribution to predict the next letter. For example, the Gaussian distribution can indicate a high probability for the letter "n" because it is common, while it can indicate a lower probability for the letter "z" which is significantly less common in the English language.

[0106] The language module 214 can select a particular key from one or more first candidate keys (e.g., "s") determined by the spatial module 212 and one or more second candidate keys (e.g., "e") determined by the language module 214 as indicated by the touch input. The one or more first candidate keys can be the same as the one or more second candidate keys, or they can be different. In some embodiments, if there is a conflict between the one or more first candidate keys and the one or more second candidate keys, the language module 214 can select a particular key based on the candidate key that is most likely to form a possible word.

[0107] In some embodiments, the language module 214 can perform post-correction on one or more words based on replacing characters with spaces or replacing spaces with characters using a language model. For example, in the case of two words "mor" and "ing", the language module 214 can perform post-correction to insert the letter "n" and create a single word "morning" from the two words. In this example, the post-correction automatically corrects the typing error caused by incorrectly interpreting the touch of "n" as a touch of the space bar by automatically inserting the letter "n", which results in two separate words "mor" and "ing".

[0108] The user interface module 216 generates graphical data to display a user interface including a virtual keyboard. In some embodiments, the user interface module 216 includes an instruction set that can be executed by the processor 202 to generate graphical data for displaying a user interface. In some embodiments, the user interface module 216 is stored in the memory 204 of the computing device 200 and can be accessed and executed by the processor 202.

[0109] In some embodiments, user interface module 216 generates graphical data for displaying a user interface, such as a virtual keyboard as part of an application, such as a messaging application. User interface module 216 may receive instructions from language module 214 to display specific keys as indicated by touch input from a user.

[0110] Flowchart 700

[0111] Figure 7 is an example flowchart 700 for selecting a specific key for a virtual keyboard according to some embodiments described herein. Flowchart 700 may be Figure 2 The computing device 200 in the embodiment of the present invention may execute, for example, Figure 1 The mobile device 115 is shown in FIG.

[0112] The method 700 starts at block 702. At block 702, a touch input by a user at a location on a virtual keyboard is detected. Block 702 may be followed by block 704.

[0113] At block 704, the spatial model is applied to map the touch input to one or more first candidate keys of the virtual keyboard. The spatial model maps the touch input to the one or more first candidate keys based on a heat map of previous touch inputs. Block 704 may be followed by block 706.

[0114] At block 706, the language model is applied to map the touch input to one or more second candidate keys of the virtual keyboard. The language model maps the touch input to one or more second candidate keys based on comparing the position of the touch input to one or more predicted next keys. Block 706 may be followed by block 708.

[0115] At box 708, a particular key of the one or more first and second candidate keys is selected as indicated by the touch input. In some embodiments, the spatial model associates each candidate key with a confidence score (e.g., a spatial score), and the language model (e.g., a language score) associates each candidate key with a confidence score. The confidence scores may also be associated with corresponding weights (e.g., based on the frequency of letters). A particular key may be selected based on a combination of confidence scores and a selection of a confidence score indicating the most likely selection. In some embodiments, a particular key is selected based on a combined confidence score exceeding a threshold.

[0116] Although referenced Figure 7 The method 700 is described in each box in FIG. 1 , but it is understood that the techniques described in the present disclosure may be performed without executing Figure 7 In some embodiments, Figure 7 One or more of the blocks shown in may be merged.

[0117] Although the specification has been described with respect to specific embodiments thereof, these specific implementations are merely illustrative and not restrictive. The concepts illustrated in the examples may be applied to other examples and embodiments.

[0118] Based on the above description, the user may be provided with controls to allow the user to make a selection regarding whether and when the system, operation or feature described herein may be able to perform user information (e.g., information about the user's data entry via a virtual keyboard, the user's touch pattern on a touch screen device, the user's keyboard language, the user's virtual keyboard settings, the user's preferences, or the user's current location), and whether the user is sent content or communications from a server. In addition, certain data may be processed in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be summarized (such as summarized to the level of a city, zip code, or state) in the case of obtaining location information, so that the user's specific location cannot be determined. Therefore, the user can control what information about the user is collected, how the information is used, and what information is provided to the user.

[0119] Note that, as known to those skilled in the art, the function blocks, operations, features, methods, devices and systems in the present disclosure can be integrated or divided into different combinations of systems, devices and function blocks. Any suitable programming language and programming techniques can be used to implement the routine of a specific embodiment. Different programming techniques can be adopted, such as procedural or object-oriented. The routine can be performed on a single processing device or multiple processors. Although steps, operations or calculations can be presented in a specific order, the order can vary in different specific embodiments. In certain embodiments, multiple steps or operations shown as continuous in the specification can be performed simultaneously.

Claims

1. A computer-implemented method for selecting a specific key, the method comprising: detecting a touch input by a user at a position on the virtual keyboard; Applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys, and wherein mapping the touch input to the one or more first candidate keys comprises: applying a global key center offset to the touch input based on a heat map of previous touch inputs proximate to the key center; applying a button level center offset to the touch input; and Mapping the touch input to the one or more first candidate keys based on the global key center offset and the key level center offset; applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on comparing the position of the touch input to one or more predicted next keys; selecting a specific key among the one or more first candidate keys and the one or more second candidate keys as indicated by the touch input; and A visual appearance of at least a portion of the virtual keyboard is adjusted based on the heat map.

2. The method according to claim 1, wherein: The spatial model determines the global key center offset based on the heat map of previous touch inputs proximate to the key center using a first Gaussian distribution.

3. The method according to claim 2, wherein: The spatial model uses a second Gaussian distribution to determine the key-level center offset.

4. The method according to claim 1, wherein: The spatial model provides one or more of: a global level correction where all keys in a keyboard layout share one parameter, a cluster level correction where all keys in the same cluster share one parameter, and a key level correction where each key has its own Gaussian parameter.

5. The method according to claim 1, wherein: Adjusting the visual appearance of the at least a portion of the virtual keyboard includes adjusting the visual appearance of a space bar to not include a visual outline.

6. The method according to claim 5, wherein: Adjusting the visual appearance of the at least a portion of the virtual keyboard includes adjusting the visual appearance of a space bar to at least expand a space above or below the space bar.

7. The method according to claim 1, further comprising: determining one or more words including the characters corresponding to the particular key; as well as The one or more words are modified based on the language model to add or remove spaces.

8. The method according to claim 1, further comprising: updating the heat map to discard previous touches older than a predetermined amount of time; as well as The visual appearance of specific keys of the virtual keyboard is updated based on the updated heat map.

9. The method according to any one of claims 1 to 8, further comprising: determining an identity of the user providing the touch input, wherein the identity is determined based on authentication input provided by the user; and The heat map corresponding to the user is retrieved.

10. A computing device for selecting a specific key, comprising: processor; and a memory coupled to the processor, the memory having instructions stored thereon, the instructions, when executed by the processor, causing the processor to perform operations comprising: detecting a touch input by a user at a position on the virtual keyboard; Applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys, and wherein mapping the touch input to the one or more first candidate keys comprises: applying a global key center offset to the touch input based on a heat map of previous touch inputs proximate to the key center; applying a button level center offset to the touch input; and Mapping the touch input to the one or more first candidate keys based on the global key center offset and the key level center offset; applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on comparing the position of the touch input to one or more predicted next keys; selecting a specific key among the one or more first candidate keys and the one or more second candidate keys as indicated by the touch input; and A visual appearance of at least a portion of the virtual keyboard is adjusted based on the heat map.

11. The computing device of claim 10, wherein: The spatial model determines the global key center offset based on the heat map of previous touch inputs proximate to the key center using a first Gaussian distribution.

12. The computing device of claim 11, wherein: The spatial model uses a second Gaussian distribution to determine the key-level center offset.

13. The computing device of claim 10, wherein: The spatial model provides one or more of: a global level correction where all keys in a keyboard layout share one parameter, a cluster level correction where all keys in the same cluster share one parameter, and a key level correction where each key has its own Gaussian parameter.

14. The computing device according to any one of claims 10 to 13, wherein: Adjusting the visual appearance of the at least a portion of the virtual keyboard includes adjusting the visual appearance of a space bar to not include a visual outline.

15. The computing device of claim 14, wherein: Adjusting the visual appearance of the at least a portion of the virtual keyboard includes adjusting the visual appearance of a space bar to at least expand a space above or below the space bar.

16. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more computers, causing the one or more computers to perform operations comprising: detecting a touch input by a user at a position on the virtual keyboard; Applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys, and wherein mapping the touch input to the one or more first candidate keys comprises: applying a global key center offset to the touch input based on a heat map of previous touch inputs proximate to the key center; applying a button level center offset to the touch input; and Mapping the touch input to the one or more first candidate keys based on the global key center offset and the key level center offset; applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on comparing the position of the touch input to one or more predicted next keys; selecting a specific key among the one or more first candidate keys and the one or more second candidate keys as indicated by the touch input; and A visual appearance of at least a portion of the virtual keyboard is adjusted based on the heat map.

17. The non-transitory computer readable medium of claim 16, wherein: The spatial model determines the global key center offset based on the heat map of previous touch inputs proximate to the key center using a first Gaussian distribution.

18. The non-transitory computer readable medium of claim 17, wherein: The spatial model uses a second Gaussian distribution to determine the key-level center offset.

19. The non-transitory computer readable medium according to any one of claims 16 to 18, wherein: Adjusting the visual appearance of the at least a portion of the virtual keyboard includes adjusting the visual appearance of a space bar to not include a visual outline.

20. The non-transitory computer readable medium of claim 19, wherein: Adjusting the visual appearance of the at least a portion of the virtual keyboard includes adjusting the visual appearance of a space bar to at least expand a space above or below the space bar.

21. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 9.

Images