A predictive input interface with improved robustness for handling low-precision input

Abstract

The present disclosure is directed to prediction and assistance techniques using a multi-region graphical keyboard interface. In particular, the system can present, on a display of a computing device, a graphical keyboard having a plurality of key regions. The plurality of key regions can include a first key region having a first set of keys and a second key region having a second set of keys. Alternatively, the system can receive a first input selecting a first selection region from the plurality of key regions. Further, the system can determine a first suggestion and a second suggestion based at least in part on the first input. Further, in response to the first input, the system can present, on a display of the computing device, an updated graphical keyboard having the plurality of key regions and a suggestion region. The suggestion region includes the first suggestion and the second suggestion.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Application No. 63 / 344,214, which has a filing date of May 20, 2022, and is incorporated by reference herein in its entirety.

[0002] TECHNICAL FIELD This disclosure relates generally to systems and methods for receiving user input with limited precision user interaction, and more particularly, to predictive and assistive techniques using multi-region graphical keyboard interfaces. [Background technology]

[0003] Computing devices can perform many tasks and provide a variety of functions. Access to such functions often requires interaction with the computing device(s) through an input interface. Various types of interfaces allow users to interact with and control such devices. Some interfaces may include numerous input options arranged such that poor accuracy of input may lead to errors, or even complete malfunction, of the computing device. Summary of the Invention

[0004] Aspects and advantages of embodiments of the disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0005] One exemplary aspect of the present disclosure is directed to a computer-implemented method. The method includes presenting, on a display of a computing device, a graphical keyboard having a plurality of key regions. The plurality of key regions includes a first key region having a first set of keys and a second key region having a second set of keys. The method further includes receiving, by the computing device, a first input selecting a first selection region from the plurality of key regions. The method further includes determining, by the computing device, a first suggestion and a second suggestion based at least in part on the first input. The method further includes presenting, on a display of the computing device in response to the first input, an updated graphical keyboard having the plurality of key regions and a suggestion region. The suggestion region may include the first suggestion and the second suggestion.

[0006] In some implementations, each key of the first set of keys and each key of the second set of keys corresponds to a symbol. In some implementations, the corresponding symbol is a glyph. A glyph can be a single representation of a character. In some implementations, a glyph is a letter of the alphabet, a number, a mark for pronouncing a letter, and / or a punctuation mark. In some implementations, a glyph is a grapheme. A grapheme can be one or more characters that represent the sound of a word.

[0007] In some implementations, each key of the graphical keyboard is assigned to one of a first key region and a second key region.

[0008] In some implementations, the method may further include receiving, by the computing device, a second input selecting a second selection region from the plurality of key regions. Additionally, the method may include determining, by the computing device, updated suggestions based at least in part on the first input and the second input. Additionally, the method may include presenting the updated suggestions in a suggestion region of the updated graphical keyboard.

[0009] In some implementations, the updated suggestions can be determined based on a ranking of the plurality of words, the updated suggestions being suggested words that exactly match the first input and the second input. Additionally, the plurality of words can be further ranked based on previous user interactions with the plurality of words.

[0010] In some implementations, the updated suggestions may be suggested words that partially match the first input and the second input, where the suggested words begin with characters (e.g., letters, numbers, punctuation marks) associated with the first input and the second input.

[0011] In some implementations, the updated suggestion can be a suggested phrase that partially matches the first input and the second input, the suggested phrase beginning with characters associated with the first input and the second input.

[0012] In some implementations, the updated graphical keyboard may include a sequence area. The method may further include determining a first decision symbol from the plurality of symbols based on the first selection area. The plurality of symbols may include a first symbol corresponding to the first key area and a second symbol corresponding to the second key area. Furthermore, the method may include presenting a decision symbol in the sequence area of ​​the updated graphical keyboard. Furthermore, the first selection area may be a first key area, and the method may further include receiving, by the computing device, a second input selecting the second key area. Furthermore, the method may include determining, by the computing device based on the second input, a second decision symbol from the plurality of symbols. The second decision symbol may be different from the first decision symbol. Thereafter, the method may include presenting a second decision symbol in the sequence area of ​​the updated graphical keyboard. In some instances, the first decision symbol is a first shape of a first color, and the second decision symbol is a first shape of a second color.

[0013] In some implementations, the multiple key regions can include a third key region having a third set of keys. The third set of keys can be different from the first set of keys and the second set of keys.

[0014] In some implementations, each key of the graphical keyboard is assigned to one of a first key region, a second key region, and a third key region.

[0015] In some implementations, the multiple key regions can include a fourth key region having a fourth set of keys. For example, the first set of keys can be keys located in the upper left quadrant of the graphical keyboard, the second set of keys can be keys located in the upper right quadrant of the graphical keyboard, the third set of keys can be keys located in the lower left quadrant of the graphical keyboard, and the fourth set of keys can be keys located in the lower right quadrant of the graphical keyboard.

[0016] In some implementations, each key of the graphical keyboard is assigned to one of a first key region, a second key region, a third key region, and a fourth key region.

[0017] In some implementations, the first set of keys may be keys located in a left row of a graphical keyboard, and the second set of keys may be keys located in a right row of the graphical keyboard.

[0018] In some implementations, the computing device may include a first physical button and a second physical button, and the first input is received by a user pressing the first physical button or the second physical button.

[0019] In some implementations, the method can further include receiving a gesture by a sensor coupled to the computing device. Further, in response to the gesture, the method can include selecting the first suggestion.

[0020] In some implementations, the first key region can have a greater number of keys than the second key region.

[0021] In some implementations, the first suggestion is a word and the second suggestion is a phrase.

[0022] Another exemplary aspect of the present disclosure is directed to a system. The system includes one or more processors and a memory storing instructions that, when executed by the processor(s), cause the system to perform operations. The operations include presenting, on a display of the computing device, a graphical keyboard having a plurality of key regions. The plurality of key regions may include a first key region having a first set of keys and a second key region having a second set of keys. Alternatively, the operations include receiving a first input selecting a first selection region from the plurality of key regions. Further, the operations include determining a first suggestion and a second suggestion based at least in part on the first input. Further, in response to the first input, the operations include presenting, on a display of the computing device, an updated graphical keyboard having the plurality of key regions and a suggestion region. The suggestion region may include the first suggestion and the second suggestion.

[0023] In some implementations, the updated graphical keyboard may include a sequence region. The operations may further include receiving a second input selecting a second selection region from the plurality of key regions. Furthermore, the operations may include determining an updated suggestion based at least in part on the first input and the second input. Furthermore, the operations may include determining a first decision symbol associated with the first input from the plurality of symbols based on the first input. The plurality of symbols may include a first symbol corresponding to the first key region and a second symbol corresponding to the second key region. Furthermore, the operations may include determining a second decision symbol associated with the second input from the plurality of symbols based on the second input. Subsequently, the operations may include presenting the first decision symbol and the second decision symbol in the sequence region of the updated graphical keyboard and presenting the updated suggestion in the suggestion region of the updated graphical keyboard.

[0024] Further examples of the present disclosure are directed to one or more non-transitory computer-readable media. The non-transitory computer-readable media may include instructions that, when executed by one or more computing devices, cause the computing device(s) to perform the operations. The operations include presenting, on a display of the computing device, a graphical keyboard having a plurality of key regions, the plurality of key regions including a first key region having a first set of keys and a second key region having a second set of keys. Alternatively, the operations include receiving a first input selecting a first selection region from the plurality of key regions. Furthermore, the operations include determining a first suggestion and a second suggestion based at least in part on the first input. In response to the first input, the operations include presenting, on a display of the computing device, an updated graphical keyboard having a plurality of key regions and a suggestion region, the suggestion region including the first suggestion and the second suggestion.

[0025] A further example of the present disclosure is directed to a computing system. The system includes one or more processors and one or more memory devices storing instructions executable by the one or more processors to perform operations. The operations include rendering, at a display component, a graphical array of input features. The graphical array of input features includes a first plurality of input features associated with a first region and a second plurality of input features associated with a second region. Additionally, the operations include determining, based on the one or more inputs received from the input component, a region sequence encoding that describes a sequence including one or more selections of the first region or the second region. Additionally, the operations include generating, based on the region sequence encoding, one or more suggested inputs.

[0026] In some implementations, the input features may correspond to linguistic symbols, and the one or more suggested inputs include suggested words of the language.

[0027] In some implementations, the first region can correspond to a first region of a graphical keyboard and the second region corresponds to a second region of the graphical keyboard.

[0028] In some implementations, the one or more inputs may include inputs associated with input signals assigned to a first region and a second region, respectively.

[0029] In some implementations, the input signal may correspond to one or more physical toggles.

[0030] In some implementations, each of the input signals may correspond to an area of ​​the touch screen that overlies the first area and the second area, respectively.

[0031] In some implementations, each of the input signals corresponds to a peripheral component.

[0032] In some implementations, generating one or more suggested inputs based on the region sequence encoding includes inputting the region sequence encoding to a machine-learned model and generating the one or more suggested inputs using the machine-learned model.

[0033] In some implementations, generating the one or more suggested inputs using the machine-learned model includes generating a probability distribution corresponding to the one or more suggested inputs.

[0034] In some implementations, the machine learned model includes a natural language model.

[0035] In some embodiments, the machine-learned model includes one or more Transformer architectures.

[0036] In some implementations, the machine-learned model can be trained on an unsupervised data set based on a sequence of regions associated with input of symbols corresponding to words of a vocabulary for a target keyboard layout.

[0037] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0038] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the detailed description, serve to explain associated principles.

[0039] Detailed descriptions of embodiments directed to those skilled in the art are set forth herein with reference to the accompanying drawings. [Brief description of the drawings]

[0040] [Figure 1A] 1 illustrates an exemplary computing environment in accordance with an exemplary embodiment of the present disclosure. [Figure 1B] 1 illustrates an exemplary computing environment in accordance with an exemplary embodiment of the present disclosure. [Figure 1C] 1 illustrates an exemplary computing environment in accordance with an exemplary embodiment of the present disclosure. [Diagram 2] 1 illustrates an exemplary forecasting system, according to an exemplary embodiment of the present disclosure. [Figure 3A] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 3B] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 3C] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 3D] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 3E] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 3F] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 3G] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 3H] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 3I] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 3J] 1 illustrates an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure. [Figure 4]1 illustrates a flowchart diagram of an exemplary method for implementing predictive and assistive techniques using a multi-region graphical keyboard interface, according to an exemplary embodiment of the present disclosure. [Diagram 5] 1 illustrates a flowchart diagram of an exemplary method for updating suggested words and / or phrases in a suggestion area of ​​a graphical keyboard interface, according to an exemplary embodiment of the present disclosure. [Figure 6] 1 illustrates a flowchart diagram of an exemplary method for presenting symbols in a sequence area of ​​a graphical keyboard interface, according to an exemplary embodiment of the present disclosure. [Figure 7] 1 illustrates a flowchart diagram of another exemplary method for implementing predictive and assistive techniques using a graphical keyboard interface, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Exemplary embodiments according to aspects of the present disclosure relate to a computing device interface that provides improved robustness for processing low-precision inputs. For example, an exemplary interface according to aspects of the present disclosure may include associating a first group of granular input features (e.g., buttons) with a first input region of the interface and associating a second group of granular input features with a second input region of the interface. In addition to or instead of processing the selection(s) of granular input features, exemplary systems and methods according to the present disclosure may process the selection of the respective region(s) with which the selected granular input features are associated. In some examples, a sequence of selected region(s) may be used to predict a desired granular input. Advantageously, in some examples, inputs that select a region may be processed with a higher confidence than inputs that select a granular input feature due to a lower threshold precision associated with the selection of a correct region compared to the selection of a correct granular input feature. In this way, for example, the interface may leverage higher confidence region-level inputs to improve the interface's tolerance for processing selections of low-precision granular input features.

[0042] In the example of text entry, a keyboard configured according to an exemplary aspect of the present disclosure may have a first group of character keys associated with a left region of the keyboard and a second group of character keys associated with a right region of the keyboard. Due to the size of the keys relative to the size of the left and right regions, the spatial precision associated with selecting a particular key associated with a desired symbol may be much higher than the spatial precision associated with selecting the correct region associated with the desired symbol. In this way, for example, a left-right-left-left sequence may be received with greater confidence than a particular sequence of received keystrokes. The more confident sequence may then be used to predict the sequence of the desired symbol, thereby improving the robustness of the keyboard to low-precision keystrokes.

[0043] Input interfaces according to exemplary aspects of the present disclosure can provide many advantages over conventional input processing techniques, such as by improving accessibility to computing device features for differently-abled users. For example, in contrast to the presently described techniques, conventional interfaces for inputting text into computer systems can typically be accomplished using digitized variations of conventional (i.e., physical) keyboard layouts, with a single key for each character (e.g., QWERTY for English, or QWERTZ for German). These conventional interfaces assume sufficient dexterity on the part of the operating user to interact with the individual character keys to input text. However, differently-abled users (e.g., users with abilities that are permanently or temporarily determined by physical abilities, environmental constraints, clothing constraints, etc.) may not have the level of dexterity required to input text in conventional systems, making conventional interfaces difficult, if not impossible, to use such conventional systems. Furthermore, existing low-precision input forms (e.g., Morse code) typically require users to learn an entirely new paradigm of communication input.

[0044] Additionally, the exemplary interfaces described herein may facilitate interaction with and control of computing devices that have constraints for any or all users. Such constraints may be inherent. For example, some computing devices have small input interfaces, such that granular input may be infeasible (e.g., smart watches with touch screens small for fingers or styluses and limited physical space for buttons). Some computing devices may have constraints dictated by convention or convenience. For example, televisions have not traditionally been associated with full keyboards for text input, and instead have traditionally only remote controls with limited alphanumeric input options. The alphanumeric input options may be difficult to use due to the small input interface, with approximately 12 small keys on the remote control, non-QWERTY, and the need to press a button multiple times to select a letter. Similarly, game consoles are often primarily used with game input controllers, such that switching to a different input device (e.g., a full keyboard) can be inefficient and inconvenient.

[0045] In some embodiments, the user interface can include a typing system having a multi-region keyboard interface (e.g., a virtual keyboard, an on-screen keyboard, a physical keyboard) that can be used to operate a computing device. (Various examples described herein are presented in the context of symbol or other text input. However, it should be understood that the scope of the present disclosure is not limited to text input devices.) The typing system can operate a desktop computer, a laptop computer, a tablet computer, a smartphone, a wearable computing device, a virtual reality system, a smart television, a game console, or virtually any computing device. Additionally or alternatively, the typing system can enable a user to operate a computing device with peripherals such as an assistive technology interface (e.g., a head mouse, a switch control, a gesture detection device, an eye gaze detection device, an electromyographic sensor).

[0046] In some implementations, graphical keyboard interfaces may be available on computing devices such as smartphones and enable users whose situationally, contextually, or permanently limited dexterity would otherwise prevent them from entering text using a standard keyboard. For example, typing on a small touchscreen-based mobile phone keyboard may be more difficult while standing on a moving train car than while sitting in a cafe. Furthermore, the techniques described herein may enable a significant improvement in the text entry experience on computing devices that may not be inherently or traditionally suitable for the task due to size or interface constraints. For example, smartwatch screens are too small for traditional keyboards that require direct pressing, for which only cumbersome text entry alternatives exist. In another example, a virtual reality headset may have a virtual keypad that is not suitable for a handheld pointer or controller due to lack of precision. In yet another example, televisions and game consoles may have remote controllers that typically use directional navigation controls that require moving the keyboard and clicking repeatedly multiple times to enter text. In yet another example, an in-car navigation system may require the use of a large screen in an interior design to facilitate information entry.

[0047] In some implementations, word and / or phrase prediction techniques can assist with text entry into a computing system. Word and / or phrase prediction techniques can enable a user to reduce the number of required keystrokes by predicting the word or phrase that the user intends to enter. Additionally, prediction techniques can suggest words that follow the currently entered word. Additionally, prediction techniques can include other features such as spell checking, speech synthesis, and shortcuts for frequently used words.

[0048] An exemplary aspect of the present disclosure is directed to a data entry system for a computing device (e.g., operation by a user in a context of limited dexterity) that uses predictive text modeling techniques to enable a minimalist interface optimized for robustness to handle low-precision inputs. A typing system (e.g., a system for data entry) may include a multi-region (e.g., two-button, three-button, four-button) keyboard with characters (e.g., letters, punctuation marks) grouped into different regions. Symbols may be grouped into multiple regions, and each region may be presented as a graphical button or collection of buttons in a multi-region graphical keyboard interface. For example, the graphical buttons may be larger than individual key buttons to enable a larger surface area to facilitate user interaction. The typing system may also include a suggestion region (e.g., a prediction region) in which directly matching and / or predicted text entries are displayed. The suggestion region may be graphically larger than individual key buttons to facilitate user interaction. Additionally, the typing system may include a sequence region that presents a sequence of previously entered user inputs (e.g., a sequence of selected graphical button(s)) to the user.

[0049] The techniques described herein can provide several technical effects and advantages. For example, as previously indicated, the techniques described herein can present an easy-to-operate multi-region keyboard interface, particularly for users with limited dexterity. Additionally, the predictive techniques described herein predict and suggest relevant words and / or phrases, allowing users to enter data faster, thereby reducing the time spent typing, editing, and other graphical user interface interactions, thus conserving computing resources (e.g., energy, processing cycles, network bandwidth, and / or the like). One or more aspects of the graphical keyboard interface can be configured to provide access to an easy-to-use keyboard interface for users with limited dexterity.

[0050] Referring now to the drawings, exemplary embodiments of the present disclosure will be discussed in further detail.

[0051] Exemplary Devices and Systems 1A illustrates a block diagram of an example computing system 100 for implementing the assistive and predictive input techniques according to an example embodiment of the present disclosure. The system 100 can include a user computing device 102, a server computing system 130, and a training computing system 150, which can be communicatively coupled via a network 180.

[0052] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, a smart device (e.g., a smart television, smart appliances), a virtual reality system, an augmented reality system, or any other type of computing device.

[0053] The user computing device 102 may include one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0054] In some implementations, the user computing device 102 can store or include one or more predictive models 120. For example, the predictive models 120 can be or otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks), or other types of machine-learned models, including non-linear and / or linear models. The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine-learned models can leverage attention mechanisms, such as self-attention. For example, some exemplary machine-learned models can include multi-head self-attention models (e.g., transformer models). An exemplary predictive model 120 is further described in FIG. 2.

[0055] In some implementations, one or more predictive models 120 may be received from a server computing system 130 over a network 180, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, a user computing device 102 may implement multiple parallel instances of a single predictive model 120.

[0056] Additionally or alternatively, one or more predictive models 140 may be included in or otherwise stored and implemented by a server computing system 130, which communicates with the user computing device 102 according to a client-server relationship. For example, the predictive models 140 may be implemented by the server computing system 140 as part of a web service. Thus, one or more models 120 may be stored and implemented in the user computing device 102 and / or one or more models 140 may be stored and implemented in the server computing system 130.

[0057] The user computing device 102 may also include one or more user input components 122 (e.g., a graphical keyboard 124, a microphone 126, a camera 128, a light sensor, a touch sensor, a button, a switch, etc.) that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be responsible for implementing the graphical keyboard 124. Other exemplary user input components include a microphone 126, a graphical keyboard 124, a camera 128, or other means by which a user may provide user input.

[0058] The user computing device 102 may also include a graphical keyboard 124 (e.g., for use via a touch screen) that receives user input. The graphical keyboard 124 may be provided (e.g., as part of an operating system (OS), a third party application, a plug-in) to one or more user devices (e.g., a computer, a smartphone, a tablet computing device, a wearable computing device). One or more aspects of the graphical keyboard 124 may be configured to provide suggested words and / or phrases by using the predictive model(s) 120 and / or the predictive model(s) 140. The graphical keyboard 124 may be a multi-region graphical keyboard interface having multiple key regions. The multiple key regions may include a first key region, a second key region, a third key region, and a fourth key region. The first key region may have a first set of keys, the second key region may have a second set of keys, and so on.

[0059] According to aspects of the disclosure, the graphical keyboard 124 may include a graphical keyboard interface (e.g., for use via a touch screen) and may be provided to or by the user computing device 102 (e.g., as part of an operating system (OS), a third party application, a plug-in). For example, with reference to FIG. 3B, the first key area 311, the second key area 312, the suggestion area 313, and the sequence area 316 may be associated with such a graphical keyboard interface. The dynamic keyboard interface may be provided in association with one or more of the application(s) executed by the user computing device 102. For example, the graphical keyboard 124 may be associated with an application (e.g., the email application 11, the virtual keyboard application 12, the text messaging application 13, and / or the like of FIG. 1B), and a graphical keyboard interface may be provided in association with such an application, as shown in FIGS. 3A-3H.

[0060] The server computing system 130 may include one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0061] In some implementations, training computing system 130 may include or be otherwise implemented by one or more server computing devices. When server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0062] As described above, the server computing system 130 may store or otherwise include one or more predictive models 140. For example, the models 140 may be or otherwise include various machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine-learned models may leverage attention mechanisms such as self-attention. For example, some exemplary machine-learned models may include multi-headed self-attention models (e.g., transformer models). Exemplary predictive models 140 are further described in FIG. 2.

[0063] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 by interacting with a training computing system 150, which can be communicatively coupled via a network 180. The training computing system 150 can be separate from the server computing system 130 or can be part of the server computing system 130.

[0064] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0065] The training computing system 150 may include a model trainer 160 that trains the predictive models 120 and / or 140 stored on the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation of error. For example, a loss function may be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent may be used to iteratively update the parameters over several training iterations.

[0066] In some implementations, performing backpropagation of the error may include performing truncated backpropagation over time. The model trainer 160 may implement several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0067] In particular, model trainer 160 can train predictive models 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, previous user interactions with the suggestion area. Training data 162 can include selection rates associated with users selecting words or phrases presented in the suggestion area of ​​the graphical user interface.

[0068] In some implementations, if the user provides consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 against the user-specific data received from the user computing device 102. In some instances, this process may be referred to as personalizing the model.

[0069] Model trainer 160 may include computer logic utilized to provide desired functionality. Model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, model trainer 160 may include program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 may include one or more sets of computer-executable instructions stored in RAM, a hard disk, or a tangible computer-readable storage medium, such as an optical or magnetic medium.

[0070] In some implementations, predictive models 120 and / or 140 may be trained by model trainer 160 using federated learning techniques to enable user-specific or device-specific model training and global updates.

[0071] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may occur over any type of wired or wireless links, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0072] The predictive models 120 and / or 140 (e.g., machine-learned models) described herein may be used in a variety of tasks, applications, and / or use cases. In some implementations, the predictive models 120 and / or 140 may include a Transformer model. A Transformer model may be a neural network that learns context, and therefore meaning, by tracking relationships in continuous data (e.g., letters in words, words in sentences). Additionally, the predictive models 120 and / or 140 may include a recurrent neural network (RNN), such as a long short-term memory (LSTM) network. Additionally, the predictive models 120 and / or 140 may include an autoregressive model and / or a feedforward network that may be extracted from any of the models.

[0073] In some implementations, when predictive models 120 and / or 140 use a Transformer model, predictive models 120 and / or 140 can also utilize masking techniques by using an encoder and a decoder.

[0074] In some implementations, predictive models 120 and / or 140 can use a beam search algorithm. A beam search algorithm can be a heuristic search algorithm that explores a graph by expanding the most promising nodes in a limited set. Beam search can be an optimization of a best-first search that allows for reduced memory requirements (e.g., to define a set of suggested words and / or phrases).

[0075] In some implementations, the input to the predictive model(s) of the present disclosure may be text or natural language data. As another example, the predictive model(s) may process the text or natural language data to generate a predicted output. As an example, the predictive model(s) may process the natural language data to generate a language encoding output. As another example, the predictive model(s) may process the text or natural language data to generate a latent text embedding output. As another example, the predictive model(s) may process the text or natural language data to generate a transformation output. As another example, the predictive model(s) may process the text or natural language data to generate a classification output. As another example, the predictive model(s) may process the text or natural language data to generate a text segmentation output. As another example, the predictive model(s) may process the text or natural language data to generate a semantic intent output. As another example, the predictive model(s) may process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the predictive model(s) can process text or natural language data to generate a predictive output.

[0076] In some implementations, the input to the predictive model(s) of the present disclosure may be image data. The predictive model(s) may process the image data to generate an output. As an example, the predictive model(s) may process the image data to generate an image recognition output (e.g., recognition of the image data, latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the predictive model(s) may process the image data to generate an image classification output. As another example, the predictive model(s) may process the image data to generate a predicted output.

[0077] In some implementations, the input to the predictive model(s) of the present disclosure can be speech data. The predictive model(s) can process the speech data to generate an output. As an example, the predictive model(s) can process the speech data to generate a speech recognition output. As another example, the predictive model(s) can process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the predictive model(s) can process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the predictive model(s) can process the speech data to generate a predicted output. In some cases, the input includes audio data representing spoken speech and the task is a speech recognition task. The output can include a text output that is mapped to the spoken speech.

[0078] In some implementations, the input to the predictive model(s) of the present disclosure may be encoded data (e.g., a sequential representation of keyboard button input(s), e.g., a region of an input interface associated with more granular input(s), etc.). The predictive model(s) may process the encoded data to generate an output. As an example, the predictive model(s) may process the encoded data to generate words and / or phrases. As another example, the predictive model(s) may process the encoded data to generate a predicted output. For example, in some embodiments, the encoded data may be input to a language model to predict one or more natural language outputs associated with the encoded data.

[0079] In some implementations, input to the predictive model(s) of the present disclosure may be sensor data (e.g., hand and / or finger gestures captured from a camera of a computing device). The predictive model(s) may process the sensor data to generate an output. As an example, the predictive model(s) may process the sensor data to generate words and / or phrases. As another example, the predictive model(s) may process the sensor data to generate a predicted output. As another example, the machine-learned model(s) may process the sensor data to generate a detection output. For example, the predictive model(s) may process the sensor data to confirm a selection of a word or phrase to be presented in a suggestion area of ​​a graphical user interface. As another example, the predictive model(s) may process the sensor data to generate a visualization output.

[0080] For example, the graphical keyboard interface may receive a user's facial gestures as input. The facial gestures may indicate a particular task to be performed by the computing system. In some instances, the user may control a mapping action on the display of the computing system by moving the user's eyes (e.g., looking left and right, up and down, blinking, winking, or other facial gestures). In some instances, the graphical keyboard interface may be custom programmed by a developer or user to perform a particular task based on a gesture (e.g., facial gesture, hand gesture, arm gesture, foot gesture). An optical sensor (e.g., camera) of the computing device may capture the gesture. For example, a user with impaired dexterity may program the graphical keyboard to perform a task (e.g., accept a first suggested word, move a selection of suggested words right, left, up, down) based on a gesture the user can perform. In some instances, a developer may pre-program the graphical keyboard interface to perform a task based on a gesture by using pre-assigned gesture mappings. The gesture mappings may be responsive to the type of dexterity impairment a user may have. A user may select the type of dexterity impairment when initiating the graphical keyboard interface or by changing the settings of the graphical keyboard interface. The graphical keyboard interface may be programmed to perform specific actions based on various gestures based on the selection of the type of dexterity impairment.

[0081] 1A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be both trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 that personalizes the model 120 based on user-specific data.

[0082] 1B illustrates a block diagram of an exemplary computing device 10 for performing operations according to an exemplary embodiment of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0083] Computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and predictive models (e.g., predictive model 120, predictive model 140). For example, each predictive model may include a machine-learned model. Exemplary applications include an email application 11, a virtual keyboard application 12, a text messaging application 13, a dictation application, a browser application, etc.

[0084] 1B, each application may communicate with several other components of the computing device, such as, for example, one or more sensors 21 (e.g., buttons, camera), a graphical user interface 22 (e.g., a virtual keyboard), audio input 23 (e.g., a microphone), a context manager, a device state component, and / or additional components 24. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0085] 1C illustrates a block diagram of an exemplary computing device 50 for performing operations according to an exemplary embodiment of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0086] Computing device 50 includes several applications (e.g., an email application, a virtual keyboard application, a text messaging application, a web browsing application). Each application may communicate with a central intelligence layer 55. Exemplary applications include a text messaging application, an email application 51, a dictation application, a virtual keyboard application 52, a text messaging application 53, a browser application, etc. In some implementations, each application can communicate with central intelligence layer 55 (and the predictive model(s) stored therein) using an API (e.g., a common public API across all applications).

[0087] The central intelligence layer 55 may include several predictive models (e.g., machine-learned models). For example, as shown in FIG. 1C, each predictive model may be provided for each application and managed by the central intelligence layer 55. In other embodiments, two or more applications (e.g., email application 51 and text messaging application 53) may share a single predictive model. For example, in some embodiments, the central intelligence layer 55 may provide a single predictive model to all applications. In some embodiments, the central intelligence layer 55 may be included within or otherwise implemented by the operating system of the computing device 50.

[0088] The central intelligence layer 55 can communicate with a central device data layer 60. The central device data layer 60 can be a centralized repository of data for the computing device 50. As illustrated in FIG. 1C, the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors 71, a graphical user interface 72 (e.g., a virtual keyboard), audio input 72, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer 55 can communicate with each device component using an API (e.g., a private API).

[0089] Exemplary Predictive Model Deployment FIG. 2 illustrates a block diagram of an exemplary prediction system 200 according to an exemplary embodiment of the present disclosure. In some implementations, a prediction model 204 is trained to receive a set of input data 202 and determine (e.g., provide, output) suggested words and / or phrases 206. The input data 202 may be a description of a user input obtained from a multi-region graphical keyboard interface. As a result of obtaining the input data 202, the prediction model 204 may determine suggested words and / or phrases 206. The suggested words and / or phrases 206 may be presented in a suggestion area of ​​a graphical user interface. Thus, in some implementations, the prediction system 200 may include a prediction model 202 operable to predict words and / or phrases. The prediction model may be a machine-learned model as described in FIGS. 1A-1C.

[0090] In some implementations, a computing system (user computing device 102, server computing device 130, training computing device 150, computing device 10, computing device 50) can use the example prediction system 200 described in FIG. 2 to process input data 202 to determine one or more suggested words and / or phrases 206.

[0091] The computing system can access (e.g., obtain, receive) input data 202 from a multi-region graphical keyboard interface. For example, a user can use the graphical keyboard 124 to enter a first input, a second input, a third input, etc. The keys of the graphical keyboard 124 can be grouped into a number of key regions. Each input can correspond to a user selecting a region from the number of key regions. In some implementations, each input can correspond to a user selecting one of the keys of a region from the number of key regions. For example, a user can select a key or region by touching a region of a graphical user interface presenting the key or region. In other examples, a user can select a region by pressing a button (e.g., a button on a smart watch, a button on a remote control) that corresponds to a region from the number of regions. In yet another example, when using a virtual reality system, a user can select a key or region by looking at the key or region, looking at a region associated with the key or region, and using a gesture (e.g., an eye blink, a hand gesture, pressing a button on a handheld device). In still other examples, a hand or arm gesture associated with a region in the number of key regions can be used to select a region.

[0092] In some instances, the predictive model 204 can dynamically determine suggested words and / or phrases based on previous user interactions (e.g., the typing history of the user, the typing history of a subset of users) and / or based on the current context of a conversation with the user. Additionally, the predictive model 204 can predict the next key to be selected or the action to be performed by the graphical keyboard interface. The predicted key or predicted action may be highlighted (e.g., highlighted) in the graphical user interface.

[0093] Further, the predictive model 204 may be trained by the model trainer 208 using various training or learning techniques, such as, for example, backpropagation of error. For example, a loss function may be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent may be used to iteratively update the parameters over several training iterations. The model trainer 160 of FIG. 1A may be an example of the model trainer 208 of the predictive system 200.

[0094] In some implementations, the model trainer 208 can train the predictive model 204 based on a set of training data 209. The training data 209 can include the training data 162 described in FIG. 1A. Additionally, the training data 209 can include, for example, user feedback 210. The user feedback 210 can include previous user interactions with the suggested words and / or phrases 206. The user feedback 210 can include a selection rate associated with a user selecting a word or phrase presented in a suggestion area of ​​the graphical user interface.

[0095] For example, in some embodiments, the predictive model 204 may include a machine-learned model configured to receive an input sequence and generate an output sequence. For example, the predictive model 204 may include a transformer architecture (e.g., an encoder structure, a decoder structure, one or more self-attention heads, etc.). For example, the predictive model 204 may include a model configured to operate on one or more input embeddings to generate a natural language output.

[0096] For example, the pattern or sequence of the selected region(s) may provide an input encoding. For example, the input encoding may include a region sequence encoding indicating a sequence of one or more region selection(s). For example, the sequence of received inputs may include a first region selection, a second region selection, a first region selection, and a first region selection (e.g., 1-2-1-1). In this example, for example, the region sequence encoding may include an embedding indicating the pattern 1-2-1-1 (e.g., "FSFF", "1211", their numeric embeddings, etc.). The region sequence embedding may be processed by the predictive model 204 to generate one or more proposed outputs (e.g., a probability distribution of the proposed outputs) that are likely to correspond to the region sequence embedding. In this way, for example, a higher confidence region level input may be used to predict a granular level output with increased robustness for processing low precision inputs.

[0097] In some embodiments, the predictive model 204 can be trained based on a training dataset including a natural language corpus. The training can be performed in an unsupervised manner with a sequence of corresponding domains (e.g., domains of a keyboard) for a given natural language string (e.g., word), which can be automatically constructed based on the position(s) of the symbolic input of the language sequence in the domain(s). Using the corresponding sequence (or its embedding) as input, the predictive model 204 can be trained to predict the given natural language string. In this way, for example, the predictive model 204 can be trained in an unsupervised manner. Such training can be in addition to personalized training based on user interaction with the suggested domains (e.g., by selecting one or more suggested input(s) as the desired input).

[0098] Exemplary Graphical User Interface 3A-3H illustrate an exemplary graphical user interface (GUI) according to an exemplary embodiment of the present disclosure.

[0099] 3A, a GUI (e.g., graphical keyboard 300) may include a first set of keys in a first key region 301 and a second set of keys in a second key region 302. Additionally, the graphical keyboard 300 may include a suggestion region 303 having a plurality of words 304. In some implementations, the first set of keys may have a plurality of keys (e.g., 20 keys, 15 keys, 10 keys, 8 keys) and the second set of keys may have a plurality of keys.

[0100] 3B, the graphical keyboard 310 may include a first set of keys in a first key region 311 and a second set of keys in a second key region 312. Additionally, the graphical keyboard 300 may include a suggestion region 313 having a plurality of words 314 and a plurality of phrases 315. Additionally, the graphical keyboard 310 may include a sequence region 316 (also referred to herein as a breadcrumb user interface) having a sequence of symbols associated with each selection region from a previous user input. In some instances, each key region of the plurality of key regions may correspond to a unique symbol. In this example, the first key region 311 may be associated with a first symbol (e.g., no dash symbol) and the second key region 312 may be associated with a second symbol (e.g., a dash symbol with a cross-hatching). As shown in the sequence region 316, the first sequence element 317 is a second symbol, the second sequence element 318 is a first symbol, and the third sequence element 319 is a second symbol. Thus, the sequence area indicates that a first user input includes a key from the second key area 312 by displaying a second symbol, and a subsequent second user input includes a key from the first key area 311 by then displaying the first symbol, etc. As the user continues to create these sequences, breadcrumbs may be presented in the sequence area 316 as reminders of previous entries.

[0101] In some implementations, the graphical keyboard 300 may have a third key region that may be associated with a third set of keys and a third symbol that may be presented in the sequence area 316. In some implementations, the graphical keyboard 300 may have a fourth key region that may be associated with a fourth set of keys and a fourth symbol that may be presented in the sequence area 316.

[0102] In some implementations, the set of keys in the first key region, the second key region, and other key regions (e.g., third, fourth) may be minimized in the graphical user interface. For example, when a user acquires a touch-sensitive memory (e.g., which keys are in which key region), the user may not need to see different keys in different key regions. In this implementation, the regions of the graphical user interface may be used for other purposes, such as only presenting the suggestion region and / or sequence region. For example, in one embodiment, only the suggestion region may be presented in the graphical user interface of a computing device (e.g., a smart watch), without the first and second key regions. In other embodiments, only the suggestion region and sequence region may be presented in the graphical user interface of a computing device.

[0103] In some embodiments, the region(s) may be associated with an input that is not in the graphical user interface. For example, the region(s) may be associated with a physical button or other input. For example, the region(s) may be associated with a button on the side of a smartwatch, a television remote, a game controller, etc., so that the graphical user interface can display suggested regions.

[0104] FIG. 3B illustrates a graphical keyboard 310 that may be presented on a smartphone, according to an exemplary embodiment. The graphical keyboard 310 may include an input area (e.g., for composing a message) and a send element that may correspond to an option to communicate data located in such input area to an application (e.g., email application 10 of FIG. 1B). The locations of the first key area 311, the second key area 312, the suggestion area 313, and the sequence area 316 of the GUI may be modified based on a user's requirements or a design choice by a developer. In this example, as shown in the sequence area, the user enters a right button-left button-right button-right button sequence (i.e., RLRRR, or shaded dash / white dash / shaded dash / shaded dash / shaded dash), which may cause the system to predict the suggested words "hello," "Kelly," "people," etc. The user may select one of the suggested words 314 or suggested phrases 315 presented in the suggestion area 313.

[0105] 3C-3E, graphical keyboards 320, 330, 340 illustrate an example of a typing process according to an exemplary embodiment of the present disclosure. The graphical keyboard 320 of FIG. 3C is a two-button keyboard having a first set of keys in a first key region 321 and a second set of keys in a second key region 322. As depicted in this example, the first key region 321 may have more characters (e.g., letters) than the second key region 322, which may be a design choice by the developer of the graphical keyboard 320.

[0106] As the user continues to type on the graphical keyboard 330 of FIG. 3D, the sequence area 331 and the suggestion area 332 can be updated with each user input. As shown in the sequence area 331, the last four user inputs are RRRL (i.e., shaded dash / shaded dash / shaded dash / open dash), and a predictive model (e.g., predictive model 204) can determine suggested words and phrases to be presented in the suggestion area 332 of the graphical keyboard 330.

[0107] 3E, a user may select the desired word "long" from a suggestion area 342 of a graphical keyboard 340. Based on the word selection, the words and / or phrases in the suggestion area 342 are updated by using a predictive model (e.g., predictive model 204).

[0108] The predictive model may determine the suggested words based on the machine learning techniques described herein. In this example, the suggested words may be ranked and presented based on the ranking of each word. In some instances, the words that match the input sequence exactly (e.g., for an input RLRR, the words in the predictive dictionary that match the input pattern RLRR) may be ranked highest and presented at the top of the suggestion area. If the ranking of two words is the same, the more frequently used word may be ranked higher. Additionally, partial matches of the input sequence (e.g., for an input RLRR, the ranked matches of longer words beginning with RLRR) may be presented in the suggestion area after the exact matching words. Additionally, phrase matches of the input sequence (e.g., for an input RLRR, the ranked matches of phrases beginning with RLRR) may also be presented in the suggestion area. In some implementations, the highest ranked words and / or phrases may be presented at the top of the suggestion area.

[0109] In some implementations, word and phrase suggestions can be paged if the top predictions (e.g., suggestions) do not match the user's intended input. For example, a region(s) of the keyboard or other input device can be used to create signals to the predictive model, as well as explicit or direct sources of text input. Additionally, the keyboard can maintain an existing, familiar keyboard layout, so that the user does not need to learn a complex new text encoding scheme.

[0110] With reference to FIG. 3F, the graphical keyboard can be implemented in different types of computing systems, such as a smart watch 350. Further, the computing system can include a first button 351, a second button 352, and the like. In some implementations, a user can select a key region from a plurality of key regions by pressing the first button 351 or the second button 352. In some implementations, when the graphical keyboard includes a third key region (e.g., as shown in FIG. 3I), a user can select the third region by pressing a third button (not shown). In some implementations, when the graphical keyboard includes a fourth key region (e.g., as shown in FIG. 3J), a user can select the fourth region by pressing a fourth button (not shown). Additionally or alternatively, a user can select a key region from a plurality of key regions by touching (e.g., pressing) an area of ​​the watch face presenting a corresponding key region. For example, a user can touch the left side of the clock face to select a first key region, or a user can touch the right side of the touch screen (e.g., clock face) to select a second key region. Additionally, the graphical keyboard can include a suggestion region (not shown) that presents suggested words and / or phrases. A user can select a suggested word or phrase by pressing an area of ​​the touch screen (e.g., clock face) that presents the suggested word or phrase. Additionally, in some instances, a user can select a suggested word or phrase that is highlighted by pressing a button (e.g., first button 351, second button 352), by pressing a button combination (e.g., first button 351 and second button 352), or by a hand gesture.

[0111] In some implementations, the graphical keyboard can be presented on a wearable computing device (e.g., a smart watch) that can be worn, for example, on a user's wrist. The wearable computing device can include a housing defining a cavity. Additionally, the wearable computing device can include button(s) disposed partially within a recess defined by an exterior surface of the housing. The button(s) can be disposed on a periphery (e.g., left side, right side, edge) of the housing. Additionally, the button(s) can include a number of sensors (e.g., strain sensors, ultrasonic sensors, motion sensors, optical sensors, and force sensors) configured to detect actuation of the button via an input provided by a user and whether a user is touching the button. The wearable computing device can include one or more processors disposed within the cavity. The wearable computing device can include a printed circuit board disposed within the cavity. The computing device 100 can further include a battery (not shown) disposed within the cavity. Additionally, the computing device can include a motion sensor disposed within the cavity of the housing. For example, the motion sensor may also include an accelerometer that may be used to capture motion data indicative of movement of the wearable computing device. Alternatively or additionally, the motion sensor may also include a gyroscope that may be used to capture motion information regarding the wearable computing device. A hand or arm gesture may be determined based on the motion data obtained from the motion sensor. The wearable computing device may include a display screen presenting a plurality of key regions of a graphical keyboard. As previously mentioned, the user input may include pressing (e.g., touching) an area of ​​the display screen associated with a key region of the plurality of key regions.

[0112] 3G, the graphical keyboard may be implemented in a smart television 360. In some implementations, the computing system may include a remote control 361 having multiple buttons. For example, the multiple key regions may correspond to different buttons on the remote control 361. A user may select a key region from the multiple key regions by the user pressing a button on the remote control 361.

[0113] 3H, a graphical keyboard may be implemented on a tablet 370. In some implementations, a first key region 371 may be separated from a second key region 372 by a predetermined distance to make it easier for a user to navigate the GUI. For example, by placing the key regions closer to the edges of the tablet 370, it may be easier for a user to type while holding the tablet with both hands.

[0114] 3I, the graphical keyboard 380 may be implemented using a three-region keyboard. The graphical keyboard 380 may have a first key region 381, a second key region 382, ​​and a third key region 383. Additionally, the graphical keyboard 380 may include a sequence region 384 that presents a first symbol associated with a selection of the first key region, a second symbol associated with a selection of the second key region, and a third symbol associated with a selection of the third key region. Additionally, the graphical keyboard 380 may include a suggestion region 385 having suggested words and / or phrases based on the selection of the key regions.

[0115] 3J, the graphical keyboard 390 may be implemented using a four-region keyboard. The graphical keyboard 390 may have a first key region 391, a second key region 392, a third key region 393, and a fourth key region 394. Additionally, the graphical keyboard 390 may include a sequence region 395 that presents a first symbol associated with a selection of the first key region, a second symbol associated with a selection of the second key region, a third symbol associated with a selection of the third key region, and a fourth symbol associated with a selection of the fourth key region. Additionally, the graphical keyboard 390 may include a suggestion region 396 having suggested words and / or phrases based on the selection of the key regions.

[0116] Exemplary Methods 4 illustrates a flow chart diagram of an exemplary method for implementing predictive and assistive techniques using a multi-region graphical keyboard interface, according to an exemplary embodiment of the present disclosure. Although FIG. 4 illustrates steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the specifically depicted order or arrangement. Various steps of method 400 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0117] At 402, a system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) can present, on a display of a computing device, a graphical keyboard having multiple key regions. The multiple key regions include a first key region having a first set of keys and a second key region having a second set of keys. For example, the first set of keys can be multiple keys of a standard keyboard. The second set of keys can be multiple keys of a standard keyboard that are different from the first set of keys.

[0118] In some implementations, the first set of keys can have multiple keys (e.g., 20 keys, 15 keys, 10 keys, 8 keys) and the second set of keys can have multiple keys. Furthermore, a key from the first set of keys requires only one user input. For example, a key can be selected by a user pressing an area associated with the first set of keys only once (e.g., touching an area associated with the first set of keys on a graphical user interface, pressing a physical button on a smartwatch or remote control). The predictive model 120 and / or 140 can determine a key from the first set of keys based on a sequence of inputs received from a user.

[0119] In some implementations, the graphical keyboard may include a QWERTY layout for English, or a QWERTZ layout for German.

[0120] In some implementations, the first set of keys may be keys located in a left row of the graphical keyboard, and the second set of keys may be keys located in a right row of the graphical keyboard.

[0121] In some implementations, the first key region has a greater number of keys than the second key region. For example, based on design studies, having a different number of keys in each key region can make a graphical keyboard interface more accessible and easier to operate.

[0122] In some implementations, the multiple key regions can include a third key region having a third set of keys, the third set of keys being different from the first set of keys and the second set of keys.

[0123] In some implementations, the plurality of key regions includes a fourth key region having a fourth set of keys, for example, the first set of keys may be keys located in an upper left quadrant of the graphical keyboard, the second set of keys may be keys located in an upper right quadrant of the graphical keyboard, the third set of keys may be keys located in a lower left quadrant of the graphical keyboard, and the fourth set of keys may be keys located in a lower right quadrant of the graphical keyboard.

[0124] At 404, the system may receive a first input selecting a first selection region from the plurality of key regions. For example, a user may select the first selection region by touching a region of the graphical keyboard in either the first key region or the second key region.

[0125] In some implementations, the computing device includes a first button and a second button, and the first input may be received by a user pressing the first button or the second button.

[0126] At 406, the system may determine a first proposal and a second proposal based at least in part on the first input. For example, the first and second proposals may be determined using the predictive model 204 using techniques described herein.

[0127] In some implementations, the first suggestion may be a word and the second suggestion may be a phrase.

[0128] At 408, in response to the first input, the system can present, on a display of the computing device, an updated graphical keyboard having a plurality of key regions and a suggestion region. The suggestion region can include a first suggestion and a second suggestion. The first suggestion or the second suggestion can be selected with a user input. When a suggestion is selected, the selected suggestion can be presented in a different region of the graphical keyboard interface.

[0129] In some implementations, the system can receive the gesture via a sensor coupled to the computing device, and in response to the gesture, the system can select the first suggestion.

[0130] 5 illustrates a flow chart diagram of an exemplary method for updating suggested words and / or phrases in a suggestion area of ​​a graphical keyboard interface, according to an exemplary embodiment of the present disclosure. Although FIG. 5 illustrates steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the specifically depicted order or arrangement. Various steps of method 500 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0131] At 502, continuing with method 400, a system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) may receive a second input selecting a second selection region from the plurality of key regions. For example, after operation 404 of method 400, the system may receive the second input.

[0132] At 504, the system can rank a plurality of words associated with the first input and the second input based at least in part on the first input and the second input. For example, a predictive model (e.g., predictive model 204) can rank the plurality of words based on the first input and the second input.

[0133] In some implementations, the plurality of words are further ranked based on previous user interactions with the plurality of words. In some implementations, the plurality of words are also ranked with previous user interactions with the suggested domain. For example, words that are frequently selected by users may be ranked higher than words that are less frequently selected by users.

[0134] At 506, the system can determine updated suggestions from the plurality of words based on the ranking of the plurality of words. As described above, a predictive model (e.g., predictive model 204) can determine the updated suggestions.

[0135] At 508, the system may present the updated suggestions in the suggestions area of ​​the updated graphical keyboard.

[0136] In some implementations, the updated suggestions may be suggested words that are an exact match of the first input and the second input.

[0137] In some implementations, the updated suggestions are suggested words that partially match the first input and the second input, for example, the suggested words can start with characters (e.g., letters) associated with the first input and the second input.

[0138] In some implementations, the updated suggestions can be suggested phrases that partially match the first input and the second input, where the suggested phrases begin with characters (e.g., letters) associated with the first input and the second input.

[0139] 6 illustrates a flow chart diagram of an exemplary method for presenting symbols in a sequence region of a graphical keyboard interface, according to an exemplary embodiment of the present disclosure. Although FIG. 6 illustrates steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the specifically depicted order or arrangement. Various steps of method 600 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0140] In some implementations, the graphical keyboard interface may include a sequence area.

[0141] At 602, a system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) can create sequence regions in the updated graphical keyboard. Examples of suggested regions are shown in FIGS. 3A-3I.

[0142] At 604, the system can determine a first determination symbol from the plurality of symbols based on the first selection region. The plurality of symbols can include a first symbol (e.g., a cross-hatched dash) corresponding to the first key region and a second symbol (e.g., a white dash) corresponding to the second key region. In one example, the first selection region can be the first key region.

[0143] At 606, the system may receive a second input selecting a second key region.

[0144] At 608, the system may determine a second decision symbol from the plurality of symbols based on the second input. In this example, the second decision symbol may be different from the first decision symbol because the second selection region may be a second key region.

[0145] At 610, the system may present the first decision symbol and the second decision symbol in a sequence area of ​​the updated graphical keyboard.

[0146] In some implementations, the first decision symbol may be a first shape of a first color (e.g., a shaded dash) and the second decision symbol is a first shape of a second color (e.g., a white dash).

[0147] 7 illustrates a flow chart diagram of another exemplary method for performing predictive and assistive techniques using a graphical keyboard interface, according to an exemplary embodiment of the present disclosure. Although FIG. 7 illustrates steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the order or arrangement specifically shown. Various steps of method 700 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0148] At 702, a system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) can render on a display component a graphical array of input features having a first plurality of input features associated with a first region and a second plurality of input features associated with a second region.

[0149] In some implementations, the input features may correspond to linguistic symbols.

[0150] In some implementations, the first region may correspond to a first region of a graphical keyboard and the second region may correspond to a second region of the graphical keyboard.

[0151] At 704, the input data can determine a region sequence encoding that describes a sequence including one or more selections of the first region or the second region based on one or more inputs received from the input component. For example, the input data 202 of FIG. 2 can be an example of one or more inputs received from the input component.

[0152] In some implementations, the one or more inputs may include inputs associated with input signals assigned to the first and second regions, respectively. The input signals may correspond to one or more physical toggles. Additionally or alternatively, each of the input signals may correspond to an area of ​​a touch screen overlying each of the first and second regions. Additionally or alternatively, each of the input signals may correspond to a peripheral component.

[0153] At 706, the system can generate one or more suggested inputs based on the region sequence encoding. For example, the graphical user interfaces illustrated in Figures 3A-3H show one or more suggested inputs generated at 706.

[0154] In some implementations, the one or more suggested inputs may include suggested words for the language (e.g., suggested words and / or suggested phrases generated by the predictive model(s) 204 in FIG. 2).

[0155] In some implementations, the one or more suggested inputs generated in 706 may be generated by inputting the region sequence encoding into a predictive model (e.g., predictive model 204, a machine-learned model) and using the machine-learned model to generate the one or more suggested inputs.

[0156] In some implementations, the one or more proposed inputs generated in 706 may be generated by generating a probability distribution that corresponds to the one or more proposed inputs.

[0157] In some implementations, the predictive model (e.g., predictive model 204, the machine-learned model in method 700) may include a natural language model. Additionally or alternatively, the predictive model (e.g., the machine-learned model) may include one or more Transformer architectures.

[0158] In some implementations, a predictive model (e.g., a machine-learned model) can be trained on an unsupervised dataset based on a sequence of regions associated with input of symbols corresponding to words in a vocabulary for a target keyboard layout.

[0159] Additional Disclosures The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions performed on and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes discussed herein may be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0160] Although the subject matter of the present disclosure has been described in detail with respect to various specific and exemplary embodiments thereof, each example is provided for the purpose of illustration and not for the purpose of limiting the present disclosure. Those skilled in the art, upon achieving the foregoing understanding, can easily create modifications, variations, and equivalents to such embodiments. Thus, the disclosure of the subject matter does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used in other embodiments to produce yet other embodiments. Thus, the present disclosure is intended to cover such modifications, variations, and equivalents.

[0161] The steps shown and / or described are merely exemplary and may be omitted, combined, and / or performed in an order other than that shown and / or described. The numbering of the steps as shown is for ease of reference only and does not imply that any particular order is required or preferred.

[0162] The functions and / or steps described herein may be embodied in computer usable data and / or computer executable instructions executed by one or more computers and / or other devices to perform one or more functions described herein. Generally, such data and / or instructions include routines, programs, objects, components, data structures, etc., that perform particular tasks and / or implement particular data types when executed by one or more processors of the computers and / or other data processing devices. The computer executable instructions may be stored on a computer readable medium, such as a hard disk, optical disk, removable storage media, solid state memory, read only memory (ROM), random access memory (RAM), etc. As will be appreciated, the functionality of such instructions may be combined and / or distributed as desired. In addition, the functionality may be embodied in whole or in part in firmware and / or hardware equivalents, such as integrated circuits, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. Certain data structures may be used to more effectively implement one or more aspects of the present disclosure, and such data structures are contemplated as being within the scope of the computer-executable instructions and / or computer-usable data described herein.

[0163] Although not required, those skilled in the art will appreciate that various aspects described herein may be embodied as methods, systems, apparatus, and / or one or more computer-readable mediums storing computer-executable instructions. Thus, aspects may take the form of entirely hardware embodiments, entirely software embodiments, entirely firmware embodiments, and / or embodiments combining software, hardware, and / or firmware aspects in any combination.

[0164] As described herein, the various methods and acts may be operable across one or more computing devices and / or networks. Functions may be distributed in any manner or may be located on a single computing device (e.g., a server, a client computer, a user device, etc.).

[0165] Aspects of the present disclosure have been described with respect to exemplary embodiments thereof. Numerous other embodiments, modifications, and / or variations within the scope and spirit of the appended claims may occur to those skilled in the art from consideration of the present disclosure. For example, one skilled in the art may recognize that the steps shown and / or described may be performed in a sequence other than that recited, and / or that one or more of the illustrated steps may be optional and / or may be combined. Any features in the following claims may be combined and / or rearranged in any possible manner.

[0166] Although the subject matter of the present disclosure has been described in detail with respect to various specific and exemplary embodiments thereof, each example is provided for the purpose of illustration and not for the purpose of limiting the present disclosure. Those skilled in the art, upon achieving the foregoing understanding, can easily create modifications, variations, and / or equivalents to such embodiments. Thus, the disclosure of the subject matter does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to those skilled in the art. For example, features illustrated and / or described as part of one embodiment may be used in other embodiments to produce yet other embodiments. Thus, the present disclosure is intended to cover such modifications, variations, and / or equivalents.

Claims

1. 1. A computer-implemented method, comprising: presenting, on a display of a computing device, a graphical keyboard having a plurality of key regions, the plurality of key regions including a first key region having a first set of keys and a second key region having a second set of keys, the first set of keys having a plurality of keys; receiving, by the computing device, a first input selecting a first selection region from the plurality of key regions; determining, by the computing device, a first suggestion and a second suggestion based at least in part on the first input; and presenting, in response to the first input, on the display of the computing device an updated graphical keyboard having the plurality of key regions and a suggestion region, the suggestion region including the first suggestion and the second suggestion; 4. A computer-implemented method comprising:

2. receiving, by the computing device, a second input selecting a second selection region from the plurality of key regions; determining, by the computing device, updated suggestions based at least in part on the first input and the second input; and presenting the updated suggestions in the suggestions area of ​​the updated graphical keyboard; The computer-implemented method of claim 1 , further comprising:

3. the updated suggestions are determined based on a ranking of a plurality of words; The computer-implemented method of claim 2 , wherein the updated suggestions are suggested words that exactly match the first input and the second input.

4. The computer-implemented method of claim 3 , wherein the plurality of words is further ranked based on previous user interactions with the plurality of words.

5. the updated suggestions being suggested words that partially match the first input and the second input; The computer-implemented method of claim 3 , wherein the suggested words begin with characters associated with the first input and the second input.

6. the updated suggestions being suggested phrases that partially match the first input and the second input; The computer-implemented method of claim 3 , wherein the suggested phrases begin with characters associated with the first input and the second input.

7. the updated graphical keyboard includes a sequence area, and the method further comprises: determining a first determination symbol from a plurality of symbols based on the first selection region, the plurality of symbols including a first symbol corresponding to the first key region and a second symbol corresponding to the second key region; and presenting the decision symbol in the sequence area of ​​the updated graphical keyboard; The computer-implemented method of claim 1 , further comprising:

8. the first selection region is the first key region, and the method further comprises: receiving, by the computing device, a second input selecting the second key region; determining, by the computing device, a second decision symbol from the plurality of symbols based on the second input, the second decision symbol being different from the first decision symbol; and presenting the second decision symbol in the sequence area of ​​the updated graphical keyboard; The computer-implemented method of claim 1 , further comprising:

9. 9. The computer-implemented method of claim 8, wherein the first decision symbol is a first shape in a first color and the second decision symbol is the first shape in a second color.

10. the plurality of key regions includes a third key region having a third set of keys; The computer-implemented method of claim 1 , wherein the third set of keys is different from the first set of keys and the second set of keys.

11. The computer-implemented method of claim 10 , wherein the plurality of key regions includes a fourth key region having a fourth set of keys.

12. the first set of keys being located in the upper left quadrant of the graphical keyboard; the second set of keys being located in the upper right quadrant of the graphical keyboard; the third set of keys being located in the lower left quadrant of the graphical keyboard; 12. The computer-implemented method of claim 11, wherein the fourth set of keys are keys located in a lower right quadrant of the graphical keyboard.

13. the first set of keys being located in a left row of the graphical keyboard; The computer-implemented method of claim 1 , wherein the second set of keys are keys located in a right row of the graphical keyboard.

14. 14. The computer-implemented method of claim 1, wherein the computing device includes a first button and a second button, and the first input is received by a user pressing the first button or the second button.

15. receiving a gesture by a sensor coupled to the computing device; and selecting the first suggestion in response to the gesture; The computer-implemented method of claim 1 , further comprising:

16. 16. The computer-implemented method of claim 1, wherein the first key region has a greater number of keys than the second key region.

17. 17. The computer-implemented method of claim 1, wherein the first suggestion is a word and the second suggestion is a phrase.

18. 1. A computing device comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations; and wherein the operation comprises: presenting on a display of the computing device a graphical keyboard having a plurality of key regions, the plurality of key regions including a first key region having a first set of keys and a second key region having a second set of keys; receiving a first input selecting a first selection region from the plurality of key regions; determining a first proposal and a second proposal based at least in part on the first input; and presenting, in response to the first input, on the display of the computing device an updated graphical keyboard having the plurality of key regions and a suggestion region, the suggestion region including the first suggestion and the second suggestion; a computing device.

19. the updated graphical keyboard includes a sequence area, and the action comprises: receiving a second input selecting a second selection region from the plurality of key regions; determining updated suggestions based at least in part on the first input and the second input; determining, based on the first input, a first decision symbol associated with the first input from a plurality of symbols, the plurality of symbols including a first symbol corresponding to the first key region and a second symbol corresponding to the second key region; determining a second decision symbol associated with the second input from the plurality of symbols based on the second input; presenting the first decision symbol and the second decision symbol in the sequence area of ​​the updated graphical keyboard; and presenting the updated suggestions in the suggestions area of ​​the updated graphical keyboard; 20. The computing device of claim 18, further comprising:

20. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations including: presenting, on a display of a computing device, a graphical keyboard having a plurality of key regions, the plurality of key regions including a first key region having a first set of keys and a second key region having a second set of keys; receiving a first input selecting a first selection region from the plurality of key regions; determining a first proposal and a second proposal based at least in part on the first input; and presenting, in response to the first input, on the display of the computing device an updated graphical keyboard having the plurality of key regions and a suggestion region, the suggestion region including the first suggestion and the second suggestion; [0023] In one or more non-transitory computer readable media,

21. 1. A computing system comprising: one or more processors; and one or more memory devices storing instructions executable to cause the one or more processors to perform operations; and the operation comprises: Rendering, at a display component, a graphical array of input features, the graphical array of input features comprising: a first plurality of input features associated with the first region; a second plurality of input features associated with the second region; and Rendering a graphical array of input features, including: determining a region sequence encoding that describes a sequence including one or more selections of the first region or the second region based on one or more inputs received from an input component; generating one or more suggested inputs based on the region sequence encoding; A computing system comprising:

22. 22. The system of claim 21, wherein the input features correspond to linguistic symbols and the one or more suggested inputs include suggested words of a language.

23. the first area corresponds to a first area of ​​a graphical keyboard; 23. The system of claim 21 or 22, wherein the second area corresponds to a second area of ​​the graphical keyboard.

24. 24. A system according to any one of claims 21 to 23, wherein the one or more inputs include inputs associated with input signals respectively assigned to the first and second domains.

25. 25. The system of claim 24, wherein the input signal corresponds to one or more physical toggles.

26. 25. The system of claim 24, wherein each of the input signals corresponds to an area of ​​a touch screen that overlies each of the first area and the second area.

27. 27. The system of claim 24, wherein each of the input signals corresponds to a peripheral component.

28. Generating one or more suggested inputs based on the region sequence encoding includes: inputting the region sequence encoding into a machine-learned model; and using the machine-learned model to generate the one or more suggested inputs; 28. The system of any one of claims 21 to 27, comprising:

29. generating the one or more suggested inputs using the machine-learned model, generating a probability distribution corresponding to the one or more proposed inputs; 30. The system of claim 28, comprising:

30. 30. The system of claim 28, wherein the machine learned model comprises a natural language model.

31. 31. The system of claim 28, wherein the machine-learned model comprises one or more Transformer architectures.

32. 32. The system of claim 28, wherein the machine-learned model is trained on an unsupervised dataset based on a sequence of regions associated with input of symbols corresponding to words of a vocabulary for a target keyboard layout.

Images