Predictive input interface with improved robustness for handling low-accuracy inputs

Abstract

This disclosure relates to prediction and assistance techniques using a multi-region graphical keyboard interface. Specifically, the system can display a graphical keyboard with multiple key regions on a display of a computing device. The multiple key regions may include a first key region having a first set of keys and a second key region having a second set of keys. Additionally, the system can receive a first input selecting a first selection region from the multiple key regions. Furthermore, the system can determine a first suggestion and a second suggestion based at least in part on the first input. Additionally, in response to the first input, the system can display an updated graphical keyboard with multiple key regions and suggestion regions on the display of the computing device. The suggestion regions include the first suggestion and the second suggestion.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Application No. 63 / 344,214, filed on May 20, 2022, which is hereby incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to systems and methods for receiving user input with limited accuracy in user interaction. More specifically, this disclosure relates to prediction and assistance techniques using a multi-region graphical keyboard interface. Background Technology

[0004] Computing devices can perform many tasks and offer a variety of functionalities. Accessing these functionalities often involves interacting with the computing device through an input interface. Different types of interfaces enable users to interact with and control such devices. Some interfaces may include numerous input options arranged such that low-accuracy input may cause the computing device to malfunction or even become completely inoperable. Summary of the Invention

[0005] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0006] One example aspect of this disclosure relates to a computer-implemented method. The method includes presenting a graphical keyboard having multiple key areas on a display of a computing device. The multiple key areas include a first key area having a first set of keys and a second key area having a second set of keys. Additionally, the method includes receiving, by the computing device, a first input selecting a first selection area from the multiple key areas. Furthermore, the method includes determining a first suggestion and a second suggestion by the computing device based at least in part on the first input. Additionally, the method includes presenting an updated graphical keyboard having multiple key areas and a suggestion area on the display of the computing device and in response to the first input. The suggestion area may include the first suggestion and the second suggestion.

[0007] In some implementations, each key in the first key set corresponds to a symbol in the second key set. In some implementations, the corresponding symbol is a glyph. A glyph can be a single representation of a character. In some implementations, a glyph is a letter, digit, letter phonogram, and / or punctuation mark from the alphabet. In some implementations, a glyph is a morpheme. A morpheme can be a letter or multiple letters representing the sound of a word.

[0008] In some implementations, each key in a graphical keyboard is assigned to one of a first key area and a second key area.

[0009] In some implementations, the method may further include receiving a second input from a plurality of key areas to select a second selection area. Additionally, the method may include determining an updated suggestion by the computing device based at least in part on the first and second inputs. Furthermore, the method may include presenting the updated suggestion in an updated suggestion area of ​​the graphical keyboard.

[0010] In some implementations, updated suggestions can be determined based on the ranking of multiple words, where the updated suggestions are suggested words that exactly match the first and second inputs. Alternatively, the multiple words can be further ranked based on previous user interactions with them.

[0011] In some implementations, the updated suggestion can be a suggested word that matches the first input and the second input, and the suggested word begins with a character (e.g., a letter, number, punctuation mark) associated with the first input and the second input.

[0012] In some implementations, the updated suggestion can be a suggestion phrase that matches the first input and the second input, and the suggestion phrase begins with a character associated with the first input and the second input.

[0013] In some implementations, the updated graphical keyboard may include a sequence of regions. The method may further include determining a first defined symbol from a plurality of symbols based on a first selection region. The plurality of symbols may include a first symbol corresponding to a first key region and a second symbol corresponding to a second key region. Additionally, the method may include presenting the defined symbol in the sequence of regions of the updated graphical keyboard. Furthermore, the first selection region may be a first key region, and the method may further include receiving a second input from a computing device that selects a second key region. Additionally, the method may include determining a second defined symbol from a plurality of symbols based on the second input. The second defined symbol may be different from the first defined symbol. Subsequently, the method may include presenting the second defined symbol in the sequence of regions of the updated graphical keyboard. In some cases, the first defined symbol is a first shape having a first color, and the second defined symbol is a first shape having a second color.

[0014] In some implementations, multiple key regions may include a third key region having a third key set. The third key set may differ from the first key set and the second key set.

[0015] In some implementations, each key in a graphical keyboard is assigned to one of a first key area, a second key area, and a third key area.

[0016] In some implementations, multiple key regions may include a fourth key region having a fourth set of keys. For example, the first set of keys may be keys located in the upper left quadrant of a graphical keyboard, the second set of keys may be keys located in the upper right quadrant of a graphical keyboard, the third set of keys may be keys located in the lower left quadrant of a graphical keyboard, and the fourth set of keys may be keys located in the lower right quadrant of a graphical keyboard.

[0017] In some implementations, each key in a graphical keyboard is assigned to one of a first key area, a second key area, a third key area, and a fourth key area.

[0018] In some implementations, the first set of keys can be the keys located in the left column of the graphical keyboard, and the second set of keys can be the keys located in the right column of the graphical keyboard.

[0019] In some implementations, the computer device may include a first physical button and a second physical button. Alternatively, the first input is received by the user pressing either the first or second physical button.

[0020] In some implementations, the method may further include receiving attitude via sensors coupled to a computing device. Additionally, in response to attitude, the method may include selecting a first suggestion.

[0021] In some implementations, the first key region may have a larger number of keys than the second key region.

[0022] In some implementations, the first suggestion is words, and the second suggestion is phrases.

[0023] Another exemplary aspect of this disclosure relates to a system. The system includes one or more processors and a memory storing instructions that, when executed by the processor, cause the system to perform operations. The operations include presenting a graphical keyboard with multiple key areas on a display of a computing device. The multiple key areas may include a first key area having a first set of keys and a second key area having a second set of keys. Additionally, the operations include receiving a first input selecting a first selection area from the multiple key areas. Furthermore, the operations include determining a first suggestion and a second suggestion based at least in part on the first input. Additionally, in response to the first input, the operations include presenting an updated graphical keyboard with multiple key areas and suggestion areas on the display of the computing device. The suggestion area may include the first suggestion and the second suggestion.

[0024] In some implementations, the updated graphical keyboard may include a sequence area. The operation may further include receiving a second input selecting a second selection area from a plurality of key areas. Additionally, the operation may include determining an updated suggestion based at least in part on the first and second inputs. Furthermore, the operation may include determining a first determined symbol associated with the first input from a plurality of symbols based on the first input. The plurality of symbols may include a first symbol corresponding to a first key area and a second symbol corresponding to a second key area. Additionally, the operation may include determining a second determined symbol associated with the second input from a plurality of symbols based on the second input. Subsequently, the operation may include presenting the first and second determined symbols in the sequence area of ​​the updated graphical keyboard, and presenting the updated suggestion in the suggestion area of ​​the updated graphical keyboard.

[0025] Further examples of this disclosure relate to one or more non-transitory computer-readable media. The non-transitory computer-readable medium may include instructions that, when executed by one or more computing devices, cause the computing device to perform an operation. The operation includes presenting a graphical keyboard with multiple key areas on a display of the computing device, wherein the multiple key areas include a first key area having a first set of keys and a second key area having a second set of keys. Additionally, the operation includes receiving a first input selecting a first selection area from the multiple key areas. Furthermore, the operation includes determining a first suggestion and a second suggestion based at least in part on the first input. In response to the first input, the operation includes presenting an updated graphical keyboard with multiple key areas and a suggestion area on a display of the computing device, wherein the suggestion area includes the first suggestion and the second suggestion.

[0026] A further example of this disclosure is directed to a computing system. The system includes: one or more processors; and one or more memory devices storing instructions executable to cause the one or more processors to perform an operation. The operation includes rendering a graphical array of input features on a display component. The graphical array of input features includes a first plurality of input features associated with a first region and a second plurality of input features associated with a second region. Additionally, the operation includes determining a region sequence encoding based on one or more inputs received from the input component, the region sequence encoding describing a sequence including one or more selections of the first region or the second region. Furthermore, the operation includes generating one or more suggested inputs based on the region sequence encoding.

[0027] In some implementations, the input features may correspond to linguistic symbols, and one or more of the suggested inputs may include suggested words of the language.

[0028] In some implementations, the first region may correspond to the first region of the graphical keyboard, and the second region may correspond to the second region of the graphical keyboard.

[0029] In some implementations, one or more inputs may include inputs associated with input signals respectively assigned to the first region and the second region.

[0030] In some implementations, the input signal may correspond to one or more physical switching switches.

[0031] In some implementations, the input signal may correspond to an area on the touchscreen that covers a first region and a second region, respectively.

[0032] In some implementations, the input signals correspond to the peripheral components.

[0033] In some implementations, generating one or more suggestion inputs based on region sequence encoding includes: inputting region sequence encodings into a machine learning model; and using the machine learning model to generate one or more suggestion inputs.

[0034] In some implementations, using a machine learning model to generate one or more suggested inputs includes generating a probability distribution corresponding to one or more suggested inputs.

[0035] In some implementations, machine learning models include natural language models.

[0036] In some implementations, the machine learning model includes one or more Transformer architectures.

[0037] In some implementations, the machine learning model can be trained on an unsupervised dataset based on a sequence of regions associated with inputs of symbols corresponding to words in a vocabulary.

[0038] Other aspects of this disclosure relate to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.

[0039] These and other features, aspects, and advantages of the various embodiments of this disclosure will become better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0040] Referring to the accompanying drawings, a detailed discussion of embodiments is set forth in this specification for those skilled in the art, in which:

[0041] Figures 1A to 1C An example computing environment according to an example embodiment of the present disclosure is described;

[0042] Figure 2An example prediction system according to an example embodiment of the present disclosure is described;

[0043] Figures 3A to 3J An example graphical user interface (GUI) according to an example embodiment of the present disclosure is depicted;

[0044] Figure 4 A flowchart is depicted illustrating an example method for performing prediction and assistance techniques using a multi-region graphical keyboard interface according to an example embodiment of the present disclosure;

[0045] Figure 5 A flowchart is provided illustrating an example method for updating suggested words and / or phrases in the suggestion area of ​​a graphical keyboard interface according to an example embodiment of the present disclosure;

[0046] Figure 6 A flowchart depicts an example method for presenting symbols in a sequence area of ​​a graphical keyboard interface according to an example embodiment of the present disclosure; and

[0047] Figure 7 A flowchart is depicted illustrating another example method for performing prediction and assistance techniques using a graphical keyboard interface according to an example embodiment of the present disclosure. Detailed Implementation

[0048] Example embodiments of aspects of this disclosure relate to computing device interfaces that provide improved robustness for processing low-accuracy inputs. For example, an example interface according to aspects of this disclosure may include associating a first set of granular input features (e.g., buttons) with a first input region of the interface, and associating a second set of granular input features with a second input region of the interface. As a supplement to or alternative to processing the selection of granular input features, example systems and methods according to this disclosure may process the selection of a corresponding region, with the selected granular input feature associated with that corresponding region. In some examples, a sequence of selected regions can be used to predict a desired granular input. Advantageously, in some examples, inputs with selected regions can be processed with higher confidence than inputs with selected granular input features, because the accuracy threshold associated with selecting the correct region is lower than that with selecting the correct granular input feature. In this way, for example, the interface may utilize region-level inputs with higher confidence to improve interface tolerance for processing low-accuracy selection of granular input features.

[0049] In the text input example, a keyboard configured according to an example aspect of this disclosure may have a first set of letter keys associated with the left region of the keyboard and a second set of letter keys associated with the right region of the keyboard. Since the key size is related to the size of the left and right regions, the spatial accuracy associated with selecting a specific key associated with the desired symbol can be much higher than the spatial accuracy associated with selecting the correct region associated with the desired symbol. In this way, for example, a left-right-left-left sequence can be received with higher confidence than a specific sequence of keystrokes received. The higher-confidence sequence can then be used to predict the desired symbol sequence, thereby improving the keyboard's robustness to low-accuracy key input.

[0050] The input interfaces of the exemplary aspects of this disclosure offer several advantages over conventional input processing techniques, such as improved accessibility to the functionality of computing devices for users with varying abilities. For example, compared to the techniques currently described, conventional interfaces for inputting text into computer systems are typically implemented using digital variations of traditional (i.e., physical) keyboard layouts (e.g., the QWERTY layout for English or the QWERTZ layout for German). These conventional interfaces assume that the user has sufficient flexibility to interact with the individual character keys in order to input text. However, users with varying abilities (e.g., whose abilities are permanently or temporarily determined by physical capabilities, environmental constraints, clothing constraints, etc.) may not possess the necessary level of flexibility to input text on conventional systems, making such conventional interfaces difficult to use (if possible). Furthermore, existing low-accuracy input formats (e.g., Morse code) often require users to learn entirely new input communication paradigms.

[0051] Additionally, the example interfaces described herein facilitate interaction with and control of computing devices that are constrained for any or all users. Such constraints may be inherent. For example, some computing devices have small input interfaces, making granular input impractical (e.g., the touchscreens of smartwatches are small relative to fingers or styluses, and physical space for buttons is limited). Some computing devices may have constraints determined by convention or convenience. For instance, televisions are traditionally not associated with full keyboards for text input, but rather with remote controls that have limited alphanumeric input options. Due to the small input interface, the remote control has approximately a dozen small keys, a non-QWERTY layout, and requires multiple button presses to select letters, making alphanumeric input options difficult to use. Similarly, game consoles are often used primarily with game input controllers, making switching to different input devices (e.g., full keyboards) inefficient and inconvenient.

[0052] In some embodiments, the user interface may include a typing system with a multi-region keyboard interface (e.g., a virtual keyboard, an on-screen keyboard, a physical keyboard) that can be used to operate a computing device. (The various examples described herein are presented in the context of symbol or other text input. However, it should be understood that the scope of this disclosure is not limited to text input devices.) The typing system can operate a desktop computer, laptop computer, tablet computer, smartphone, wearable computing device, virtual reality system, smart TV, game console, or virtually any computing device. Additionally or alternatively, the typing system may enable a user to operate the computer device by means of peripheral devices such as assistive technology interfaces (e.g., head mouse, switch control, posture detection device, gaze detection device, electromyography sensor)

[0053] In some implementations, graphical keyboard interfaces can be used with computing devices such as smartphones, enabling users whose contexts, scenarios, or permanently limited flexibility would otherwise prevent them from using a standard keyboard to input text. For example, typing on a small touchscreen-based mobile phone keypad while standing in a bumpy train carriage might be more challenging than sitting in a coffee shop. Furthermore, the techniques described herein unlock significantly improved text input experiences on computing devices that may have been inherently or traditionally unsuitable for the task due to their size or interface constraints. For example, the screens of smartwatches are too small for traditional keyboards that require direct pressing, and for such screens, only cumbersome text input options exist. In another example, virtual reality headsets may have virtual keypads, where handheld indicators or controllers are unsuitable due to a lack of accuracy. In yet another example, televisions and game consoles may have remote controls that typically employ directional navigation controls that require multiple iterative clicks across a keyboard to input text. In yet another example, in-vehicle navigation systems may require the use of large screens in interior design to facilitate information entry.

[0054] In some implementations, word and / or phrase prediction techniques can assist text input in computational systems. By predicting the words or phrases a user intends to input, word and / or phrase prediction techniques can reduce the number of keystrokes required. Additionally, prediction techniques can suggest words after the currently being typed word. Furthermore, prediction techniques may include other features such as spell checking, speech synthesis, and shortcuts for frequently used words.

[0055] An exemplary aspect of this disclosure relates to a data entry system for a computing device that uses predictive text modeling techniques to achieve a minimal interface optimized for robustness in handling low-accuracy input (e.g., for user operation in scenarios with limited flexibility). The typing system (e.g., a system for data entry) may include a multi-region (e.g., two-button, three-button, four-button) keyboard where characters (e.g., letters, punctuation marks) are grouped into different regions. Symbols may be grouped into multiple regions, each of which may be presented as a graphical button or a set of buttons in a multi-region graphical keyboard interface. For example, graphical buttons may be larger than individual key buttons to achieve a large surface area for easier user interaction. The typing system may also include a suggestion area (e.g., a prediction area) that displays directly matched and / or predicted text entries. The suggestion area may be graphically larger than individual key buttons for easier user interaction. Additionally, the typing system may include a sequence area that presents the user with a sequence of previously entered user input (e.g., a sequence of selected graphical buttons).

[0056] The techniques described herein offer numerous technical effects and benefits. For example, as previously indicated, the techniques described herein can make multi-region keyboard interfaces more user-friendly (especially for users with flexibility constraints). Additionally, the predictive techniques described herein predict and suggest relevant words and / or phrases, which allows users to input data faster, thereby reducing the time spent on typing, editing, and other graphical user interface interactions, thus saving computational resources (e.g., energy, processing cycles, network bandwidth, etc.). One or more aspects of the graphical keyboard interface can be configured to provide users with limited flexibility with access to an easy-to-use keyboard interface.

[0057] Exemplary embodiments of this disclosure will now be discussed in further detail with reference to the accompanying drawings.

[0058] Example devices and systems

[0059] Figure 1A A block diagram of an example computing system 100 for performing assistive and predictive input techniques according to an exemplary embodiment of the present disclosure is depicted. System 100 may include a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0060] User computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, a smart device (e.g., a smart TV, a smart home appliance), a virtual reality system, an augmented reality system, or any other type of computing device.

[0061] User computing device 102 may include one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 may store data 116 and instructions 118 executed by processor 112 to cause user computing device 102 to perform operations.

[0062] In some implementations, the user computing device 102 may store or include one or more prediction models 120. For example, the prediction model 120 may be, or may otherwise include, various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models (e.g., Transformer models). Figure 2 The example prediction model 120 is further discussed in the paper.

[0063] In some implementations, one or more prediction models 120 may be received from server computing system 130 via network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instances of a single prediction model 120.

[0064] Alternatively or additionally, one or more prediction models 140 may be included in or otherwise stored and implemented by a server computing system 130, which communicates with the user computing device 102 according to a client-server relationship. For example, the prediction model 140 may be implemented by the server computing system 140 as part of a web service. Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0065] User computing device 102 may also include one or more user input components 122 (e.g., graphical keyboard 124, microphone 126, camera 128, optical sensor, touch sensor, button, switch, etc.) for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components may be used to implement graphical keyboard 124. Other example user input components include microphone 126, graphical keyboard 124, camera 128, or other means by which a user may provide user input.

[0066] User computing device 102 may also include a graphical keyboard 124 for receiving user input (e.g., for use via a touchscreen). The graphical keyboard 124 may be provided to one or more user devices (e.g., computers, smartphones, tablet computing devices, wearable computing devices) as part of an operating system (OS), third-party applications, or plugins. One or more aspects of the graphical keyboard 124 may be configured to provide suggested words and / or phrases using prediction models 120 and / or 140. The graphical keyboard 124 may be a multi-region graphical keyboard interface with multiple key areas. The multiple key areas may include a first key area, a second key area, a third key area, and a fourth key area. The first key area may have a first set of keys, the second key area may have a second set of keys, and so on.

[0067] According to aspects of this disclosure, the graphical keyboard 124 may include a graphical keyboard interface (e.g., for use via a touchscreen), which may be provided to or provided by the user computing device 102 (e.g., as part of an operating system (OS), third-party applications, or plugins). For example, see reference... Figure 3B The first key area 311, the second key area 312, the suggestion area 313, and the sequence area 316 can be associated with this graphical keyboard interface. The dynamic keyboard interface can be provided in association with one or more applications executed by the user computing device 102. For example, the graphical keyboard 124 can be associated with applications (e.g., Figure 1B (Associated with email application 11, virtual keyboard application 12, text messaging application 13, etc.) and as Figures 3A to 3H As illustrated, a graphical keyboard interface can be provided in association with such applications.

[0068] Server computing system 130 may include one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operatively connected. Memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 may store data 136 and instructions 138 executed by processor 132 to cause server computing system 130 to perform operations.

[0069] In some implementations, the server computing system 130 may include one or more server computing devices or be implemented by such one or more server computing devices in other ways. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0070] As described above, server computing system 130 may store or otherwise include one or more predictive models 140. For example, model 140 may be, or may otherwise include, various machine learning models. Example machine learning models include neural networks or other multi-layered nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models (e.g., Transformer models). Figure 2 Example prediction model 140 is further discussed in the paper.

[0071] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0072] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is otherwise implemented by said one or more server computing devices.

[0073] The training computing system 150 may include a model trainer 160 that uses various training or learning techniques, such as, for example, error backpropagation, to train prediction models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130. For example, a loss function can be used to update one or more parameters of the model via model backpropagation (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update parameters across multiple training iterations.

[0074] In some implementations, performing error backpropagation may include performing backpropagation through time with truncation. The model trainer 160 may perform various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0075] Specifically, model trainer 160 may train prediction models 120 and / or 140 based on a set of training data 162. Training data 162 may include, for example, previous user interactions with suggestion areas. Training data 162 may include selection rates associated with user selections of words or phrases presented in suggestion areas of the graphical user interface.

[0076] In some implementations, training examples may be provided by the user computing device 102 if the user has provided consent. Therefore, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as model personalization.

[0077] Model trainer 160 may include computer logic for providing desired functionality. Model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, model trainer 160 may include a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 may include a set of one or more computer-executable instructions stored in a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0078] In some implementations, prediction models 120 and / or 140 may be trained by model trainer 160 using federated learning techniques to allow for user-specific or device-specific model training and global updates.

[0079] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. Generally, communication over network 180 can be conducted using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL) via any type of wired and / or wireless connection.

[0080] The predictive models 120 and / or 140 (e.g., machine learning models) described in this specification can be used for a variety of tasks, applications, and / or use cases. In some implementations, predictive models 120 and / or 140 may include Transformer models. A Transformer model may be a neural network that learns context and therefore meaning by tracking relationships in sequence data (e.g., letters in words, words in sentences). Alternatively, predictive models 120 and / or 140 may include recurrent neural networks (RNNs), such as Long Short-Term Memory (LSTM) networks. Additionally, predictive models 120 and / or 140 may include autoregressive models and / or feedforward networks that can be distilled from any model.

[0081] In some implementations, when prediction models 120 and / or 140 use Transformer models, prediction models 120 and / or 140 can also utilize masking techniques by using encoders and decoders.

[0082] In some implementations, a beam search algorithm may be used when prediction models 120 and / or 140. A beam search algorithm can be a heuristic search algorithm that explores a graph by expanding on the most promising nodes in a finite set. A beam search can be an optimization of a best-first search (e.g., defining a set of suggested words and / or phrases) that reduces memory requirements.

[0083] In some implementations, the input to the predictive model of this disclosure can be text or natural language data. The predictive model can process the text or natural language data to generate output. As an example, the predictive model can process natural language data to generate language-encoded output. As another example, the predictive model can process text or natural language data to generate latent text embedding output. As another example, the predictive model can process text or natural language data to generate translation output. As another example, the predictive model can process text or natural language data to generate classification output. As another example, the predictive model can process text or natural language data to generate text segmentation output. As another example, the predictive model can process text or natural language data to generate semantic intent output. As another example, the predictive model can process text or natural language data to generate amplified text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As yet another example, the predictive model can process text or natural language data to generate predictive output.

[0084] In some implementations, the input to the prediction model of this disclosure can be image data. The prediction model can process the image data to generate an output. As an example, the prediction model can process image data to generate an image recognition output (e.g., recognition of image data, latent embedding of image data, encoded representation of image data, hashing of image data, etc.). As another example, the prediction model can process image data to generate an image classification output. As yet another example, the prediction model can process image data to generate a prediction output.

[0085] In some implementations, the input to the prediction model of this disclosure can be speech data. The prediction model can process the speech data to generate an output. As an example, the prediction model can process speech data to generate a speech recognition output. As another example, the prediction model can process speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the prediction model can process speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As yet another example, the prediction model can process speech data to generate a predicted output. In some cases, the input includes audio data representing spoken utterance, and the task is a speech recognition task. The output may include a text output mapped to spoken utterance.

[0086] In some implementations, the input to the predictive model of this disclosure may be encoded data (e.g., a sequence representation of keyboard button inputs, such as regions of an input interface associated with finer-grained input). The predictive model may process the encoded data to generate output. As an example, the predictive model may process the encoded data to generate words and / or phrases. As another example, the predictive model may process the encoded data to generate predicted output. For example, in some embodiments, the encoded data may be input into a language model to predict one or more natural language outputs associated with the encoded data.

[0087] In some implementations, the input to the predictive model of this disclosure may be sensor data (e.g., hand and / or finger poses captured from a camera on a computing device). The predictive model may process the sensor data to generate output. As an example, the predictive model may process the sensor data to generate words and / or phrases. As another example, the predictive model may process the sensor data to generate predictive output. As yet another example, the machine learning model may process the sensor data to generate detection output. For example, the predictive model may process the sensor data to confirm the selection of words or phrases presented in a suggestion area of ​​a graphical user interface. As yet another example, the predictive model may process the sensor data to generate visualization output.

[0088] For example, a graphical keyboard interface (GUI) can receive a user's facial gestures as input. Facial gestures can instruct the computing system to perform specific tasks. In some cases, a user can control mapped actions on the computing system's display by moving their eyes (e.g., moving them left and right, looking up and down, squinting, blinking, or other facial gestures). In other cases, the GUI can be custom-programmed by the developer or user to perform certain tasks based on gestures (e.g., facial gestures, hand gestures, arm gestures, foot gestures). The computing device's optical sensors (e.g., a camera) can capture this gesture. For example, a user with a dexterity impairment can be programmed to perform tasks based on their executable gestures (e.g., accepting a first suggested word, selecting to move the suggested word right, left, up, or down). In some cases, developers can pre-program the GUI to perform tasks based on gestures using pre-assigned gesture maps. The gesture map may depend on the type of dexterity impairment the user may have. The user can select the type of dexterity impairment when starting the GUI or by changing the GUI's settings. The graphical keyboard interface can be programmed based on the selection of the type of dexterity impairment to perform specific actions based on different postures.

[0089] Figure 1AAn example computing system that can be used to implement this disclosure is illustrated. Other computing systems may also be used. For example, in some implementations, user computing device 102 may include model trainer 160 and training dataset 162. In such implementations, model 120 may be trained locally at user computing device 102 and both may be used. In some implementations of such implementations, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0090] Figure 1B A block diagram of an example computing device 10 implemented according to an example embodiment of the present disclosure is depicted. The computing device 10 may be a user computing device or a server computing device.

[0091] The computing device 10 includes multiple applications (e.g., application 1 to application N). Each application contains its own machine learning library and prediction model (e.g., prediction model 120, prediction model 140). For example, each prediction model may include a machine learning model. Example applications include email application 11, virtual keyboard application 12, text messaging application 13, dictation application, browser application, etc.

[0092] like Figure 1B As illustrated, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors 21 (e.g., buttons, cameras), a graphical user interface 22 (e.g., a virtual keyboard), audio input 23 (e.g., a microphone), a scene manager, device state components, and / or additional components 24. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0093] Figure 1C A block diagram depicts an example computing device 50 implemented according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0094] Computing device 50 includes multiple applications (e.g., email application, virtual keyboard application, text messaging application, web browsing application). Each application can communicate with the central intelligence layer 55. Example applications include a text messaging application, email application 51, dictation application, virtual keyboard application 52, text messaging application 53, browser application, etc. In some implementations, each application can communicate with the central intelligence layer 55 (and the predictive models stored therein) using an API (e.g., a common API across all applications).

[0095] The central intelligence layer 55 may include multiple predictive models (e.g., machine learning models). For example, such as Figure 1CAs illustrated, a corresponding predictive model can be provided for each application, and these corresponding predictive models can be managed by a central intelligence layer 55. In other implementations, two or more applications (e.g., email application 51 and text messaging application 53) can share a single predictive model. For example, in some implementations, the central intelligence layer 55 can provide a single predictive model for all applications. In some implementations, the central intelligence layer 55 can be included within the operating system of the computing device 50 or otherwise implemented by the operating system of the computing device.

[0096] The central intelligence layer 55 can communicate with the central device data layer 60. The central device data layer 60 can be a centralized data storage repository for the computing device 50. For example... Figure 1C As illustrated, the central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors 71, a graphical user interface 72 (e.g., a virtual keyboard), an audio input 72, a scene manager, a device state component, and / or additional components. In some implementations, the central device data layer 55 may communicate with each device component using an API (e.g., a private API).

[0097] Example Predictive Model Deployment

[0098] Figure 2 A block diagram of an example prediction system 200 according to an exemplary embodiment of the present disclosure is depicted. In some implementations, a prediction model 204 is trained to receive a set of input data 202 and determine (e.g., provide, output) suggested words and / or phrases 206. The input data 202 may describe user input obtained from a multi-area graphical keyboard interface. As a result of obtaining the input data 202, the prediction model 204 may determine suggested words and / or phrases 206. The suggested words and / or phrases 206 may be presented on a suggestion area of ​​the graphical user interface. Therefore, in some implementations, the prediction system 200 may include a prediction model 202 operable to predict words and / or phrases. The prediction model may be as follows: Figures 1A to 1C The machine learning model described.

[0099] In some implementations, the computing system (user computing device 102, server computing device 130, training computing device 150, computing device 10, computing device 50) can use Figure 2 The example prediction system 200 described herein processes input data 202 to determine one or more suggested words and / or phrases 206.

[0100] The computing system can access (e.g., acquire, receive) input data 202 from a multi-area graphical keyboard interface. For example, a user can use the graphical keyboard 124 to input a first input, a second input, a third input, etc. The keys of the graphical keyboard 124 can be grouped into multiple key areas. Each input corresponds to a region selected by the user from the multiple key areas. In some implementations, each input may correspond to any key in the region selected by the user from the multiple key areas. For example, a user can select a key or region by touching the area that presents the key or region in the graphical user interface. In another example, a user can select a region by pressing a button corresponding to a region in the multiple areas (e.g., a button on a smartwatch, a button on a remote control). In yet another example, when using a virtual reality system, a user can select a key or region by gazing at the area associated with the key or region and using gestures (e.g., blinking, hand gestures, pressing a button on a handheld device). In yet another example, a hand or arm gesture associated with a region in the multiple key areas can be used to select the region.

[0101] In some cases, prediction model 204 may dynamically determine suggested words and / or phrases based on previous user interactions (e.g., user typing history, typing history of a subset of users) and / or based on the current context of the conversation with the user. Additionally, prediction model 204 may predict the next key to be selected or the action to be performed by the graphical keyboard interface. The predicted key or predicted action may be highlighted (e.g., shown) on the graphical user interface.

[0102] Additionally, the prediction model 204 can be trained by the model trainer 208 using various training or learning techniques, such as, for example, error backpropagation. For instance, a loss function can be used to update one or more parameters of the model via model backpropagation (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters across multiple training iterations. Figure 1A The model trainer 160 in the prediction system 200 can be an example of the model trainer 208 in the prediction system 200.

[0103] In some implementations, the model trainer 208 may train the prediction model 204 based on a set of training data 209. The training data 209 may include... Figure 1A The training data 162 described herein. Additionally, the training data 209 may include, for example, user feedback 210. User feedback 210 may include previous user interactions with suggested words and / or phrases 206. User feedback 210 may include selection rates associated with words or phrases presented in the suggestion area of ​​the graphical user interface.

[0104] For example, in some embodiments, prediction model 204 may include a machine learning model configured to receive an input sequence and generate an output sequence. For example, prediction model 204 may include a Transformer architecture (e.g., an encoder structure, a decoder structure, one or more self-attention heads, etc.). For example, prediction model 204 may include a model configured to operate on one or more input embeddings to generate natural language output.

[0105] For example, the pattern or sequence of selected regions can provide input encoding. For example, the input encoding can include region sequence encoding indicating the selection of one or more regions. For example, the received input sequence can include: selection of a first region, selection of a second region, selection of a first region, and selection of a first region (e.g., first-second-first-first). In this example, for example, the region sequence encoding can include an embedding indicating the pattern first-second-first-first (e.g., “FSFF”, “1211”, its numeric embedding, etc.). The region sequence embedding can be processed by prediction model 204 to generate one or more proposed outputs (e.g., a probability distribution of the proposed outputs) that may correspond to the region sequence embedding. For example, in this way, a higher confidence region-level input can be used to predict outputs with improved robustness for handling low-accuracy inputs.

[0106] In some embodiments, the prediction model 204 may be trained on a training dataset containing a natural language corpus. Training can be performed unsupervised by automatically constructing a sequence of regions (e.g., keyboard regions) based on the positions of symbolic inputs of the language sequence within the regions, given a natural language sequence (e.g., words). Using the corresponding sequence (or its embedding) as input, the prediction model 204 can be trained to predict the given natural language sequence. For example, the prediction model 204 can be trained unsupervised in this way. Such training can also be performed in addition to personalized training based on user interactions with the suggested regions (e.g., by selecting one or more suggested inputs as desired inputs).

[0107] Example graphical user interface

[0108] Figures 3A to 3H An example graphical user interface (GUI) according to an example embodiment of the present disclosure is depicted.

[0109] refer to Figure 3AThe GUI (e.g., a graphical keyboard 300) may include a first set of keys in a first key area 301 and a second set of keys in a second key area 302. Additionally, the graphical keyboard 300 may include a suggestion area 303 having multiple words 304. In some implementations, the first set of keys may have multiple keys (e.g., 20 keys, 15 keys, 10 keys, 8 keys), and the second set of keys may have multiple keys.

[0110] refer to Figure 3B The graphical keyboard 310 may include a first set of keys in a first key area 311 and a second set of keys in a second key area 312. Additionally, the graphical keyboard 300 may include a suggestion area 313 having multiple words 314 and multiple phrases 315. Furthermore, the graphical keyboard 310 may include a sequence area 316 (also referred to herein as a breadcrumb user interface) having a sequence of symbols associated with each selected area from previous user input. In some cases, each of the multiple key areas may correspond to a unique symbol. In this example, the first key area 311 may be associated with a first symbol (e.g., a transparent hyphen symbol), and the second key area 312 may be associated with a second symbol (e.g., a shaded hyphen symbol). As depicted in the sequence area 316, a first sequence element 317 is a second symbol, a second sequence element 318 is a first symbol, and a third sequence element 319 is a second symbol. Therefore, the sequence area indicates that the first user input includes a key from the second key area 312 by displaying a second symbol, and then indicates that the subsequent second user input includes a key from the first key area 311 by displaying a first symbol, and so on. As the user continues to create these sequences, breadcrumbs can be presented in the sequence area 316 as a reminder of the previous input.

[0111] In some implementations, the graphical keyboard 300 may have a third key area that can be associated with a third set of keys. The third key area may also be associated with a third symbol to be presented in the sequence area 316. In some implementations, the graphical keyboard 300 may have a fourth key area that can be associated with a fourth set of keys. The fourth key area may also be associated with a fourth symbol to be presented in the sequence area 316.

[0112] In some implementations, the set of keys in the first key area, second key area, and other key areas (e.g., third key area, fourth key area) can be minimized on the graphical user interface. For example, once a user acquires touch type memory (e.g., remembers which keys are in which key areas), the user may not need to view the different keys in different key areas. With this implementation, the areas of the graphical user interface can be used for other purposes, such as presenting only suggested areas and / or sequence areas. For example, in one embodiment, only suggested areas may be presented on the graphical user interface of a computing device (e.g., a smartwatch), without presenting the first and second key areas. In another embodiment, only suggested areas and sequence areas may be presented on the graphical user interface of a computing device.

[0113] In some embodiments, the area may be associated with an input not displayed on the graphical user interface. For example, the area may be associated with a corresponding physical button or other input. For instance, the area may be associated with a button on the side of a smartwatch, a TV remote, a game controller, etc., allowing the graphical user interface to display the suggested area.

[0114] Figure 3B A graphical keyboard 310, which can be displayed on a smartphone according to an example embodiment, is depicted. The graphical keyboard 310 may include an input area (e.g., for composing a message) and a sending element that can be used to transmit data located in such an input area to an application (e.g., [application / application]). Figure 1B The options in the email application 10 correspond to those in the GUI. The positions of the first key area 311, the second key area 312, the suggestion area 313, and the sequence area 316 on the GUI can be modified based on user requests or developer design choices. In this example, as depicted in the sequence area, the user enters a sequence of right button-left button-right button-right button-right button (i.e., RLRRR or shaded connector / white connector / shaded connector / shaded connector / shaded connector), based on which the system can predict suggested words such as "Hello", "Kelly", and "People". The user can choose one of the suggested words 314 or suggested phrases 315 presented in the suggestion area 313.

[0115] Now for reference Figures 3C to 3E According to exemplary embodiments of the present disclosure, graphical keyboards 320, 330, and 340 depict examples of typing processes. Figure 3C The graphical keyboard 320 is a two-button keyboard, which has a first set of keys in a first key area 321 and a second set of keys in a second key area 322. As depicted in this example, the first key area 321 may have more characters (e.g., letters) than the second key area 322, which may be a design choice for the developers of the graphical keyboard 320.

[0116] When the user continues Figure 3D When typing on the graphical keyboard 330, the sequence area 331 and the suggestion area 332 can be updated after each user input. As depicted in the sequence area 331, the last four user inputs are RRRL (i.e., shaded hyphen / shaded hyphen / shaded hyphen / white hyphen), and a prediction model (e.g., prediction model 204) can determine the suggested words and phrases presented in the suggestion area 332 of the graphical keyboard 330.

[0117] refer to Figure 3E The user can select the desired word 'long' from the suggestion area 342 of the graphical keyboard 340. Based on the word selection, the words and / or phrases in the suggestion area 342 are updated using a prediction model (e.g., prediction model 204).

[0118] The predictive model can determine suggested words based on the machine learning techniques described in this paper. In this example, suggested words can be ranked and presented based on the ranking of each word. In some cases, words that exactly match the input sequence (e.g., for input RLRR, words in the prediction dictionary that match the input pattern of RLRR) are ranked highest and presented at the top of the suggestion area. When two words have the same ranking, the more frequently used word is ranked higher. Additionally, partial matches of the input sequence (e.g., for input RLRR, rank matches of longer words starting with RLRR) can be presented in the suggestion area after the words that exactly match. Furthermore, phrase matches of the input sequence (e.g., for input RLRR, rank matches of phrases starting with RLRR) can also be presented in the suggestion area. In some implementations, the highest-ranked word and / or phrase can be presented at the top of the suggestion area.

[0119] In some implementations, word and phrase suggestions can be paginated when the top prediction (e.g., suggestion) does not match the user's intended input. For example, areas of the keyboard or other input device can be used to signal the predictive model, rather than just explicit or direct sources of text input. Additionally, the keyboard can maintain familiarity with existing keyboard layouts, eliminating the need for users to learn complex new text encoding schemes.

[0120] refer to Figure 3F The graphical keyboard can be implemented on different types of computing systems (such as a smartwatch 350). Additionally, the computing system may include a first button 351, a second button 352, etc. In some implementations, the user can select a key area from multiple key areas by pressing the first button 351 or the second button 352. In some implementations, when the graphical keyboard includes a third key area (e.g., as shown in the image), the graphical keyboard can be implemented on different types of computing systems (such as a smartwatch 350). Figure 3IWhen the graphical keyboard includes a fourth key area (as depicted in the diagram), the user can select the third area by pressing the third button (not shown). In some implementations, when the graphical keyboard includes a fourth key area (e.g., as shown in the diagram), the user can select the third area. Figure 3J When the fourth area is selected (as depicted in the diagram), the user can select the fourth area by pressing the fourth button (not shown). Alternatively or additionally, the user can select a key area from multiple key areas by touching (e.g., pressing) an area on the dial that displays the corresponding key area. For example, the user can touch the left side of the dial to select the first key area, or the user can touch the right side of the touchscreen (e.g., the dial) to select the second key area. Furthermore, the graphical keypad may include a suggestion area (not shown) that displays suggested words and / or phrases. The user can select suggested words or phrases by pressing the area on the touchscreen (e.g., the dial) that displays suggested words or phrases. Additionally, in some cases, the user can select highlighted suggested words or phrases by pressing buttons (e.g., first button 351, second button 352), by a combination of pressing buttons (e.g., first button 351 and second button 352), or by hand gestures.

[0121] In some implementations, a graphical keyboard may be presented on a wearable computing device (e.g., a smartwatch) that can be worn, for example, on a user's arm. The wearable computing device may include a housing defining a cavity. Additionally, the wearable computing device may include a button partially positioned within a recess defined by an outer surface of the housing. The button may be positioned at the periphery of the housing (e.g., left, right, or edge). Furthermore, the button may include multiple sensors (e.g., strain sensors, ultrasonic sensors, motion sensors, optical sensors, and force sensors) configured to detect button actuation via user-provided input and whether the user is touching the button. The wearable computing device may include one or more processors positioned within the cavity. The wearable computing device may include a printed circuit board disposed within the cavity. The computing device 100 may further include a battery (not shown) disposed within the cavity. Additionally, the computing device may include a motion sensor positioned within the cavity of the housing. For example, the motion sensor may also include an accelerometer that can be used to capture motion data indicating the movement of the wearable computing device. Optionally or additionally, the motion sensor may also include a gyroscope, which can also be used to capture motion information about the wearable computing device. Hand or arm posture can be determined based on motion data obtained from the motion sensor. The wearable computing device may include a display screen to present multiple key areas of a graphical keyboard. As previously discussed, user input may include pressing (e.g., touching) an area on the display screen associated with a key area among the multiple key areas.

[0122] refer to Figure 3GA graphical keyboard can be implemented on a smart TV 360. In some implementations, the computing system may include a remote control 361 with multiple buttons. For example, multiple key areas may correspond to different buttons on the remote control 361. The user can select a key area from the multiple key areas by pressing a button on the remote control 361.

[0123] refer to Figure 3H A graphical keyboard can be implemented on the tablet computer 370. In some implementations, the first key area 371 can be separated from the second key area 372 by a predetermined distance to make it easier for the user to operate the GUI. For example, by placing the key areas closer to the edge of the tablet computer 370, it is easier for the user to type when holding the tablet computer with both hands.

[0124] refer to Figure 3I The graphical keyboard 380 can be implemented using a three-zone keyboard. The graphical keyboard 380 may have a first key area 381, a second key area 382, ​​and a third key area 383. Additionally, the graphical keyboard 380 may include a sequence area 384 that presents a first symbol associated with a selection of the first key area, a second symbol associated with a selection of the second key area, and a third symbol associated with a selection of the third key area. Furthermore, the graphical keyboard 380 may include a suggestion area 385 that has suggested words and / or phrases based on the selection of the key areas.

[0125] refer to Figure 3J The graphical keyboard 390 can be implemented using a four-zone keyboard. The graphical keyboard 390 may have a first key area 391, a second key area 392, a third key area 393, and a fourth key area 394. Additionally, the graphical keyboard 390 may include a sequence area 395 that displays a first symbol associated with a selection of the first key area, a second symbol associated with a selection of the second key area, a third symbol associated with a selection of the third key area, and a fourth symbol associated with a selection of the fourth key area. Furthermore, the graphical keyboard 390 may include a suggestion area 396 that displays suggested words and / or phrases based on the selection of the key areas.

[0126] Example Method

[0127] Figure 4 A flowchart depicts an example method for performing prediction and assistance techniques using a multi-region graphical keyboard interface according to an example embodiment of this disclosure. Although Figure 4 The steps performed in a specific order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the specifically illustrated order or arrangement. The steps of method 400 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of this disclosure.

[0128] At 402, the system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) may display a graphical keyboard with multiple key areas on the display of the computing device. The multiple key areas include a first key area having a first set of keys and a second key area having a second set of keys. For example, the first set of keys may be multiple keys on a standard keyboard. The second set of keys may be multiple keys on a standard keyboard that are different from the first set of keys.

[0129] In some implementations, the first key set may have multiple keys (e.g., 20 keys, 15 keys, 10 keys, 8 keys), and the second key set may have multiple keys. Additionally, a key in the first key set can be selected with only a single user input. For example, a user can select a key by pressing (e.g., touching an area associated with the first key set in a graphical user interface, pressing a physical button on a smartwatch or remote control) the area associated with the first key set only once. Predictive models 120 and / or 140 can determine the key from the first key set based on the input sequence received from the user.

[0130] In some implementations, the graphical keyboard may include a QWERTY layout for the English language or a QWERTZ layout for the German language.

[0131] In some implementations, the first set of keys can be the keys located in the left column of the graphical keyboard, and the second set of keys can be the keys located in the right column of the graphical keyboard.

[0132] In some implementations, the first key area has a larger number of keys than the second key area. For example, based on design research, having a different number of keys in each key area can make the graphical keyboard interface more accessible and easier to operate.

[0133] In some implementations, multiple key regions may include a third key region with a third key set. This third key set is different from the first and second key sets.

[0134] In some implementations, multiple key regions include a fourth key region having a fourth set of keys. For example, the first set of keys may be keys located in the upper left quadrant of a graphical keyboard, the second set of keys may be keys located in the upper right quadrant of a graphical keyboard, the third set of keys may be keys located in the lower left quadrant of a graphical keyboard, and the fourth set of keys may be keys located in the lower right quadrant of a graphical keyboard.

[0135] At position 404, the system can receive a first input from multiple key areas to select a first selection area. For example, a user can select the first selection area by touching an area on a graphical keyboard located in either the first or second key area.

[0136] In some implementations, the computer device includes a first button and a second button, and the first input can be received by the user pressing either the first button or the second button.

[0137] At point 406, the system can determine the first and second recommendations based at least in part on the first input. For example, the first and second recommendations can be determined using a prediction model 204 employing the techniques described herein.

[0138] In some implementations, the first suggestion can be a word or phrase, while the second suggestion can be a short phrase.

[0139] At point 408, in response to the first input, the system can display an updated graphical keyboard with multiple key areas and suggestion areas on the display of the computing device. The suggestion area may include a first suggestion and a second suggestion. The first suggestion or the second suggestion can be selected by user input. Once a suggestion is selected, the selected suggestion can be displayed in different areas of the graphical keyboard interface.

[0140] In some implementations, the system can receive attitude via sensors coupled to a computing device. Alternatively, in response to the attitude, the system can select a first suggestion.

[0141] Figure 5 A flowchart depicts an example method for updating suggested words and / or phrases in the suggestion area of ​​a graphical keyboard interface according to an example embodiment of this disclosure. Although Figure 5 The steps performed in a specific order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the specifically illustrated order or arrangement. The steps of method 500 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of this disclosure.

[0142] At 502, continuing with method 400, the system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) may receive a second input selecting a second selection area from multiple key areas. For example, the system may receive the second input after operation 404 in method 400.

[0143] At point 504, the system can rank multiple words associated with the first and second inputs, at least in part, based on the first and second inputs. For example, a prediction model (e.g., prediction model 204) can rank multiple words based on the first and second inputs.

[0144] In some implementations, multiple words are further ranked based on previous user interactions with those words. In others, the ranking also considers previous user interactions with the suggestion area. For example, words frequently selected by users may rank higher than those less frequently selected.

[0145] At point 506, the system can determine updated recommendations from multiple words based on their rankings. As previously described, a predictive model (e.g., predictive model 204) can determine the updated recommendations.

[0146] At point 508, the system can display updated suggestions in the suggestion area of ​​the updated graphical keyboard.

[0147] In some implementations, the updated suggestions can be suggested words that exactly match the first and second inputs.

[0148] In some implementations, the updated suggestions are suggested words that match the first and second input portions. For example, suggested words may begin with characters (e.g., letters) associated with the first and second inputs.

[0149] In some implementations, the updated suggestion can be a suggested phrase that matches the first input and the second input, and the suggested phrase begins with a character (e.g., a letter) associated with the first input and the second input.

[0150] Figure 6 A flowchart depicts an example method for presenting symbols in a sequence area of ​​a graphical keyboard interface according to an example embodiment of the present disclosure. Although Figure 6 The steps performed in a specific order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the specifically illustrated order or arrangement. The steps of method 600 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of this disclosure.

[0151] In some implementations, the graphical keyboard interface may include a sequence of regions.

[0152] At 602, the system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) can create a sequence region in the updated graphical keyboard. Figures 3A to 3I An example of a suggested area is shown in the figure.

[0153] At 604, the system can determine a first specified symbol from a plurality of symbols based on a first selected region. The plurality of symbols may include a first symbol corresponding to a first key region (e.g., a shaded hyphen) and a second symbol corresponding to a second key region (e.g., a white hyphen). In one example, the first selected region may be the first key region.

[0154] At position 606, the system can receive a second input that selects a second key area.

[0155] At point 608, the system can determine a second determined symbol from a plurality of symbols based on a second input. In this example, the second determined symbol may differ from the first determined symbol because the second selection region may be a second key region.

[0156] At point 610, the system can present a first and a second designated symbol in the sequence area of ​​the updated graphical keyboard.

[0157] In some implementations, the first defining symbol may be a first shape with a first color (e.g., a shaded hyphen), and the second defining symbol may be a first shape with a second color (e.g., a white hyphen).

[0158] Figure 7 A flowchart depicts another example method for performing prediction and assistance techniques using a graphical keyboard interface according to an example embodiment of this disclosure. Although Figure 7 The steps performed in a specific order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the specifically illustrated order or arrangement. The steps of method 700 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of this disclosure.

[0159] At 702, the system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) can render a graphical array of input features on a display component, the graphical array of input features having: a first plurality of input features associated with a first region; and a second plurality of input features associated with a second region.

[0160] In some implementations, input features can correspond to linguistic symbols.

[0161] In some implementations, the first area may correspond to the first area of ​​the graphical keyboard, and the second area may correspond to the second area of ​​the graphical keyboard.

[0162] At 704, the system can determine a region sequence code based on one or more inputs received from the input component, the region sequence code describing a sequence including one or more selections of a first region or a second region. For example, Figure 2The input data 202 in the input data can be an example of one or more inputs received from the input component.

[0163] In some implementations, one or more inputs may include inputs associated with input signals respectively assigned to a first region and a second region. The input signals may correspond to one or more physical toggle switches. Alternatively or additionally, the input signals may correspond to areas on the touchscreen that respectively cover the first and second regions. Alternatively or additionally, the input signals may correspond to peripheral components.

[0164] At position 706, the system can generate one or more proposed inputs based on region sequence encoding. For example, Figures 3A to 3H The graphical user interface depicted illustrates one or more suggested inputs generated at point 706.

[0165] In some implementations, one or more suggested inputs may include suggested words from the language (e.g., by...). Figure 2 (Suggested words and / or suggested phrases generated by prediction model 204).

[0166] In some implementations, one or more suggested inputs generated at 706 can be generated by inputting region sequence encodings into a prediction model (e.g., prediction model 204, machine learning model) and using the machine learning model to generate one or more suggested inputs.

[0167] In some implementations, one or more suggested inputs generated at 706 can be generated by generating a probability distribution corresponding to one or more suggested inputs.

[0168] In some implementations, the predictive model (e.g., predictive model 204, the machine learning model in method 700) may include a natural language model. Alternatively, the predictive model (e.g., the machine learning model) may include one or more Transformer architectures.

[0169] In some implementations, the predictive model (e.g., a machine learning model) can be trained on an unsupervised dataset based on a region sequence associated with inputs that correspond to symbols of words in a vocabulary, for a target keyboard layout.

[0170] Additional Disclosure

[0171] This paper discusses technologies related to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and divisions of tasks and functionality among and within components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0172] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of explanation and not as a limitation of this disclosure. Modifications, alterations, and equivalents of such embodiments will be readily apparent to those skilled in the art upon understanding the foregoing. Therefore, this disclosure does not exclude such modifications, alterations, and / or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to produce further embodiments. Therefore, this disclosure is intended to cover such modifications, alterations, and equivalents.

[0173] The steps depicted and / or described are merely illustrative and may be omitted, combined, and / or performed in an order different from that depicted and / or described; the numbering of the steps depicted is merely for convenience of reference and does not imply that any particular order is necessary or preferred.

[0174] The functions and / or steps described herein may be embodied in computer-usable data and / or computer-executable instructions executed by one or more computers and / or other devices to perform one or more functions described herein. Generally, such data and / or instructions include routines, programs, objects, components, data structures, etc., that perform specific tasks and / or implement specific data types when executed by one or more processors in a computer and / or other data processing device. Computer-executable instructions may be stored in computer-readable media such as hard disks, optical disks, removable storage media, solid-state storage, read-only memory (ROM), random access memory (RAM), etc. As understood, the functionality of such instructions may be combined and / or distributed as needed. Furthermore, this functionality may be wholly or partially embodied in firmware and / or hardware equivalents such as integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. Specific data structures may be used to more efficiently implement one or more aspects of this disclosure, and such data structures are contemplated within the scope of the computer-executable instructions and / or computer-usable data described herein.

[0175] Although not essential, those skilled in the art will understand that the various aspects described herein can be embodied as methods, systems, devices, and / or one or more computer-readable media storing computer-executable instructions. Therefore, aspects can take the form of all-hardware embodiments, all-software embodiments, all-firmware embodiments, and / or embodiments combining software, hardware, and / or firmware aspects in any combination.

[0176] As described herein, various methods and actions can operate across one or more computing devices and / or networks. This functionality can be distributed in any manner or may reside within a single computing device (e.g., a server, client computer, user device, etc.).

[0177] The aspects of this disclosure have been described with reference to their illustrative embodiments. Numerous other embodiments, modifications, and / or alterations within the scope and spirit of the appended claims will arise in those skilled in the art upon careful reading of this disclosure. For example, those skilled in the art will understand that the depicted and / or described steps may be performed in a different order than those stated, and / or one or more exemplified steps may be optional and / or combined. Any and all features of the following claims may be combined and / or rearranged in any possible manner.

[0178] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of explanation and not as a limitation of this disclosure. Modifications, alterations, and / or equivalents of such embodiments will be readily apparent to those skilled in the art upon understanding the foregoing. Therefore, this disclosure does not exclude such modifications, alterations, and / or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, features illustrated and / or described as part of one embodiment may be used with another embodiment to produce further embodiments. Therefore, this disclosure is intended to cover such modifications, alterations, and / or equivalents.

Claims

1. A computer-implemented method, comprising: A graphical keyboard with multiple key areas is presented on the display of a computing device, wherein the multiple key areas include a first key area having a first set of keys and a second key area having a second set of keys, the first set of keys having multiple keys; The computing device receives a first input from the plurality of key regions to select a first selection region; A first determined symbol is determined from a plurality of symbols based on the first selected region, wherein each key region in the plurality of key regions corresponds to a unique symbol, and wherein the plurality of symbols includes a first symbol corresponding to the first key region and a second symbol corresponding to the second key region, wherein if the first selected region corresponds to the first key region, the first determined symbol is the first symbol, and if the first selected region corresponds to the second key region, the first determined symbol is the second symbol. The computing device determines the first and second recommendations based at least in part on the first input; as well as In response to the first input, an updated graphical keyboard with the plurality of key areas, sequence areas, and suggestion areas is presented on the display of the computing device, wherein the suggestion areas include the first suggestion and the second suggestion, and wherein the sequence areas include the first determined symbol. The method further includes: The computing device receives a second input from the plurality of key regions to select a second selection region; The computing device determines a second determined symbol from the plurality of symbols based on the second input; The computing device determines the updated recommendation based at least in part on the first input and the second input; The second determined symbol is presented in the sequence area of ​​the updated graphical keyboard; and The updated suggestions are presented in the suggestion area of ​​the updated graphical keyboard.

2. The computer-implemented method of claim 1, wherein the updated suggestion is determined based on the ranking of a plurality of words, and wherein the updated suggestion is a suggested word that perfectly matches the first input and the second input.

3. The computer-implemented method of claim 2, wherein the plurality of words are further ranked based on previous user interactions with the plurality of words.

4. The computer-implemented method of claim 1, wherein the updated suggestion is a suggested word that matches the first input and the second input portions, and wherein the suggested word begins with a character associated with the first input and the second input.

5. The computer-implemented method of claim 1, wherein the updated suggestion is a suggested phrase matching the first input and the second input portions, and wherein the suggested phrase begins with a character associated with the first input and the second input.

6. The computer-implemented method of claim 1, wherein the first determined symbol is a first shape having a first color, and the second determined symbol is the first shape having a second color.

7. The computer-implemented method of claim 1, wherein the plurality of key regions includes a third key region having a third key set, and wherein the third key set is different from the first key set and the second key set. The method further includes: The computing device receives a third input from the plurality of key regions to select a third selection region; The computing device determines a third determining symbol from the plurality of symbols based on the third input; The computing device determines the updated recommendation based at least in part on the first input, the second input, and the third input; The third identified symbol is presented in the sequence area of ​​the updated graphical keyboard; as well as The updated suggestions are presented in the suggestion area of ​​the updated graphical keyboard.

8. The computer-implemented method of claim 7, wherein the plurality of key regions includes a fourth key region having a fourth key set, and wherein the fourth key set is different from the first key set, the second key set, and the third key set. The method further includes: The computing device receives a fourth input from the plurality of key regions to select a fourth selection region; The computing device determines a fourth determining symbol from the plurality of symbols based on the fourth input; The computing device determines the updated recommendation based at least in part on the first input, the second input, the third input, and the fourth input; The fourth determined symbol is presented in the sequence area of ​​the updated graphical keyboard; as well as The updated suggestions are presented in the suggestion area of ​​the updated graphical keyboard.

9. The computer-implemented method of claim 8, wherein the first set of keys is located in the upper left quadrant of the graphical keyboard, wherein the second set of keys is located in the upper right quadrant of the graphical keyboard, wherein the third set of keys is located in the lower left quadrant of the graphical keyboard, and wherein the fourth set of keys is located in the lower right quadrant of the graphical keyboard.

10. The computer-implemented method of claim 1, wherein the first set of keys is the keys located in the left column of the graphical keyboard, and wherein the second set of keys is the keys located in the right column of the graphical keyboard.

11. The computer-implemented method of claim 1, wherein the computer device includes a first button and a second button, and the first input is received by a user pressing the first button or the second button.

12. The computer-implemented method of claim 1, further comprising: Attitude is received via sensors coupled to the computing device; as well as In response to the stated posture, the first suggestion is selected.

13. The computer-implemented method of claim 1, wherein the first key region has a greater number of keys than the second key region.

14. The computer-implemented method of claim 1, wherein the first suggestion is a word or phrase, and wherein the second suggestion is a phrase.

15. A computer device comprising: One or more processors; as well as The memory stores instructions that, when executed by the one or more processors, cause the computer device to perform operations, including: A graphical keyboard with multiple key areas is presented on the display of a computing device, wherein the multiple key areas include a first key area having a first set of keys and a second key area having a second set of keys; Receive a first input from the plurality of key regions to select a first selection region; A first determined symbol is determined from a plurality of symbols based on the first selected region, wherein each key region in the plurality of key regions corresponds to a unique symbol, and wherein the plurality of symbols includes a first symbol corresponding to the first key region and a second symbol corresponding to the second key region, wherein if the first selected region corresponds to the first key region, the first determined symbol is the first symbol, and if the first selected region corresponds to the second key region, the first determined symbol is the second symbol. The first and second recommendations are determined at least in part based on the first input; and In response to the first input, an updated graphical keyboard with the plurality of key areas, sequence areas, and suggestion areas is presented on the display of the computing device, wherein the suggestion areas include the first suggestion and the second suggestion, and wherein the sequence areas include the first determined symbol, wherein the operation further includes: Receive a second input that selects a second selection region from the plurality of key regions; The updated recommendation is determined at least in part based on the first input and the second input; Based on the second input, a second determined symbol associated with the second input is determined from the plurality of symbols; The first determined symbol and the second determined symbol are presented in the sequence area of ​​the updated graphical keyboard; and The updated suggestions are presented in the suggestion area of ​​the updated graphical keyboard.

16. One or more non-transitory computer-readable media, the non-transitory computer-readable media comprising instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations including: A graphical keyboard with multiple key areas is presented on the display of a computing device, wherein the multiple key areas include a first key area having a first set of keys and a second key area having a second set of keys; Receive a first input from the plurality of key regions to select a first selection region; A first determined symbol is determined from a plurality of symbols based on the first selected region, wherein each key region in the plurality of key regions corresponds to a unique symbol, and wherein the plurality of symbols includes a first symbol corresponding to the first key region and a second symbol corresponding to the second key region, wherein if the first selected region corresponds to the first key region, the first determined symbol is the first symbol, and if the first selected region corresponds to the second key region, the first determined symbol is the second symbol. The first and second recommendations are determined at least in part based on the first input; as well as In response to the first input, an updated graphical keyboard with the plurality of key areas, sequence areas, and suggestion areas is presented on the display of the computing device, wherein the suggestion areas include the first suggestion and the second suggestion, and wherein the sequence areas include the first determined symbol. The operation further includes: Receive a second input that selects a second selection region from the plurality of key regions; A second determined symbol is determined from the plurality of symbols based on the second input; The updated recommendation is determined at least in part based on the first input and the second input; The second determined symbol is presented in the sequence area of ​​the updated graphical keyboard; and The updated suggestions are presented in the suggestion area of ​​the updated graphical keyboard.

Images