Predictive input interface with improved robustness for processing low accuracy inputs

Abstract

The invention relates to prediction and assistance techniques using a multi-region graphical keyboard interface. In particular, a system may present a graphical keyboard having a plurality of key regions on a display of a computing device. The plurality of key regions may include a first key region having a first key set and a second key region having a second key set. Additionally, the system may receive a first input to select a first selection region from the plurality of key regions. Further, the system may determine a first suggestion and a second suggestion based at least in part on the first input. In addition, in response to the first input, the system may present an updated graphical keyboard having a plurality of key regions and a suggested region on a display of the computing device. The suggested area includes a first suggestion and a second suggestion.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Application No. 63 / 344,214, filed May 20, 2022, and is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to systems and methods for receiving user input with limited accuracy user interaction. More specifically, the present disclosure relates to predictive and assistive techniques using a multi-region graphical keyboard interface. Background Art

[0004] Computing devices can perform many tasks and provide a variety of functionality. Accessing such functionality often involves interacting with the computing device through an input interface. Different types of interfaces enable users to interact with and control such devices. Some interfaces may include numerous input options that are arranged so that low-accuracy inputs may cause the computing device to malfunction, or even become completely inoperable. Summary of the invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] An example aspect of the present disclosure relates to a computer-implemented method. The method includes presenting a graphical keyboard having multiple key areas on a display of a computing device. The multiple key areas include a first key area having a first key set and a second key area having a second key set. Additionally, the method includes receiving, by the computing device, a first input selecting a first selection area from the multiple key areas. Additionally, the method includes determining, by the computing device at least in part based on the first input, a first suggestion and a second suggestion. Additionally, the method includes presenting an updated graphical keyboard having multiple key areas and a suggestion area on a display of the computing device and in response to the first input. The suggestion area may include a first suggestion and a second suggestion.

[0007] In some implementations, each key in the first key set and each key in the second key set corresponds to a symbol. In some implementations, the corresponding symbol is a glyph. A glyph can be a single representation of a character. In some implementations, a glyph is a letter of the alphabet, a number, a letter pronunciation mark, and / or a punctuation mark. In some implementations, a glyph is a grapheme. A grapheme can be a letter or a plurality of letters representing the sound of a word.

[0008] In some implementations, each key in the graphical keyboard is assigned to one of the first key region and the second key region.

[0009] In some implementations, the method may further include receiving, by the computing device, a second input selecting a second selection area from the plurality of key areas. Additionally, the method may include determining, by the computing device, an updated suggestion based at least in part on the first input and the second input. Additionally, the method may include presenting the updated suggestion in a suggestion area of ​​the updated graphical keyboard.

[0010] In some implementations, updated suggestions may be determined based on the ranking of the plurality of words, and wherein the updated suggestions are suggested words that fully match the first input and the second input. Additionally, the plurality of words may be further ranked based on previous user interactions with the plurality of words.

[0011] In some implementations, the updated suggestion can be a suggested word that partially matches the first input and the second input, and where the suggested word begins with a character (e.g., a letter, a number, a punctuation mark) associated with the first input and the second input.

[0012] In some implementations, the updated suggestion can be a suggested phrase that partially matches the first input and the second input, and where the suggested phrase begins with characters associated with the first input and the second input.

[0013] In some implementations, the updated graphical keyboard may include a sequence area. The method may further include determining a first determination symbol from a plurality of symbols based on a first selection area. The plurality of symbols may include a first symbol corresponding to a first key area and a second symbol corresponding to a second key area. Additionally, the method may include presenting a determination symbol in a sequence area of ​​the updated graphical keyboard. Furthermore, the first selection area may be a first key area, and the method may further include receiving a second input selecting a second key area by a computing device. Additionally, the method may include determining a second determination symbol from a plurality of symbols based on a second input by a computing device. The second determination symbol may be different from the first determination symbol. Subsequently, the method may include presenting a second determination symbol in a sequence area of ​​the updated graphical keyboard. In some cases, the first determination symbol is a first shape having a first color, and the second determination symbol is a first shape having a second color.

[0014] In some implementations, the plurality of key regions may include a third key region having a third set of keys. The third set of keys may be different from the first set of keys and the second set of keys.

[0015] In some implementations, each key in the graphical keyboard is assigned to one of the first key region, the second key region, and the third key region.

[0016] In some implementations, the plurality of key regions may include a fourth key region having a fourth key set. For example, the first key set may be keys located in the upper left quadrant of the graphical keyboard, the second key set may be keys located in the upper right quadrant of the graphical keyboard, the third key set may be keys located in the lower left quadrant of the graphical keyboard, and the fourth key set may be keys located in the lower right quadrant of the graphical keyboard.

[0017] In some implementations, each key in the graphical keyboard is assigned to one of the first key region, the second key region, the third key region, and the fourth key region.

[0018] In some implementations, the first set of keys can be keys located in a left column of the graphical keyboard, and the second set of keys can be keys located in a right column of the graphical keyboard.

[0019] In some implementations, the computer device may include a first physical button and a second physical button. Additionally, the first input is received by a user pressing the first physical button or the second physical button.

[0020] In some implementations, the method may further include receiving a gesture via a sensor coupled to the computing device. Additionally, in response to the gesture, the method may include selecting the first suggestion.

[0021] In some implementations, the first key area can have a greater number of keys than the second key area.

[0022] In some implementations, the first suggestion is a word and the second suggestion is a phrase.

[0023] Another example aspect of the present disclosure relates to a system. The system includes one or more processors and a memory storing instructions that, when executed by the processor, cause the system to perform operations. The operations include presenting a graphical keyboard having multiple key areas on a display of a computing device. The multiple key areas may include a first key area having a first key set and a second key area having a second key set. Additionally, the operations include receiving a first input selecting a first selection area from the multiple key areas. Furthermore, the operations include determining a first suggestion and a second suggestion based at least in part on the first input. Additionally, in response to the first input, the operations include presenting an updated graphical keyboard having multiple key areas and a suggestion area on a display of the computing device. The suggestion area may include a first suggestion and a second suggestion.

[0024] In some implementations, the updated graphical keyboard may include a sequence area. The operation may further include receiving a second input selecting a second selection area from a plurality of key areas. Additionally, the operation may include determining updated suggestions based at least in part on the first input and the second input. Furthermore, the operation may include determining a first determined symbol associated with the first input from a plurality of symbols based on the first input. The plurality of symbols may include a first symbol corresponding to the first key area and a second symbol corresponding to the second key area. Additionally, the operation may include determining a second determined symbol associated with the second input from a plurality of symbols based on the second input. Subsequently, the operation may include presenting the first determined symbol and the second determined symbol in the sequence area of ​​the updated graphical keyboard, and presenting updated suggestions in the suggestion area of ​​the updated graphical keyboard.

[0025] Further examples of the present disclosure relate to one or more non-transitory computer-readable media. The non-transitory computer-readable medium may include instructions that, when executed by one or more computing devices, cause the computing device to perform an operation. The operation includes presenting a graphical keyboard having multiple key areas on a display of the computing device, wherein the multiple key areas include a first key area having a first key set and a second key area having a second key set. Additionally, the operation includes receiving a first input selecting a first selection area from the multiple key areas. Furthermore, the operation includes determining a first suggestion and a second suggestion based at least in part on the first input. In response to the first input, the operation includes presenting an updated graphical keyboard having multiple key areas and a suggestion area on a display of the computing device, wherein the suggestion area includes a first suggestion and a second suggestion.

[0026] A further example of the present disclosure is directed to a computing system. The system includes: one or more processors; and one or more memory devices storing instructions that are executable to cause the one or more processors to perform operations. The operations include rendering a graphical array of input features on a display component. The graphical array of input features includes a first plurality of input features associated with a first area and a second plurality of input features associated with a second area. Additionally, the operations include determining a region sequence encoding based on one or more inputs received from an input component, the region sequence encoding describing a sequence including one or more selections of a first area or a second area. Additionally, the operations include generating one or more suggested inputs based on the region sequence encoding.

[0027] In some implementations, the input features can correspond to language tokens, and wherein the one or more suggested inputs include suggested words of the language.

[0028] In some implementations, the first region can correspond to a first zone of a graphical keyboard, and wherein the second region corresponds to a second zone of the graphical keyboard.

[0029] In some implementations, the one or more inputs can include inputs associated with input signals assigned to the first region and the second region, respectively.

[0030] In some implementations, the input signal can correspond to one or more physical toggle switches.

[0031] In some implementations, the input signal can correspond to regions on the touch screen that respectively cover the first area and the second area.

[0032] In some implementations, the input signals correspond to peripheral components, respectively.

[0033] In some implementations, generating one or more suggestion inputs based on the region sequence encoding includes: inputting the region sequence encoding into a machine learning model; and using the machine learning model to generate the one or more suggestion inputs.

[0034] In some implementations, where using the machine learning model to generate one or more suggested inputs includes generating a probability distribution corresponding to the one or more suggested inputs.

[0035] In some implementations, the machine learning model includes a natural language model.

[0036] In some implementations, the machine learning model includes one or more Transformer architectures.

[0037] In some implementations, the machine learning model can be trained on an unsupervised dataset for a target keyboard layout based on region sequences associated with input of symbols corresponding to words in a vocabulary.

[0038] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0039] These and other features, aspects and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0041] Figures 1A to 1C An example computing environment is depicted according to an example embodiment of the present disclosure;

[0042] Figure 2An example prediction system according to an example embodiment of the present disclosure is depicted;

[0043] FIG. 3A to FIG. 3J depicts an example graphical user interface (GUI) according to an example embodiment of the present disclosure;

[0044] Figure 4 A flowchart depicting an example method of performing predictive and assistive techniques using a multi-region graphical keyboard interface according to an example embodiment of the present disclosure;

[0045] Figure 5 A flowchart depicting an example method of updating suggested words and / or phrases in a suggestion area of ​​a graphical keyboard interface according to an example embodiment of the present disclosure;

[0046] Figure 6 A flowchart depicting an example method of presenting symbols in a sequence area of ​​a graphical keyboard interface according to an example embodiment of the present disclosure; and

[0047] Figure 7 Depicted is a flow chart of another example method of performing predictive and assistive techniques using a graphical keyboard interface according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] Example embodiments according to aspects of the present disclosure relate to a computing device interface that provides improved robustness for processing low-accuracy inputs. For example, an example interface according to aspects of the present disclosure may include associating a first set of granular input features (e.g., buttons) with a first input area of ​​the interface, and associating a second set of granular input features with a second input area of ​​the interface. In addition to or in lieu of processing the selection of granular input features, example systems and methods according to the present disclosure may process the selection of corresponding areas, with the selected granular input features associated with the corresponding areas. In some examples, a sequence of selected areas may be used to predict the desired granular input. Advantageously, in some examples, the input of a selected area may be processed with a higher confidence than the input of a selected granular input feature, because the accuracy threshold associated with selecting the correct area is lower than that of selecting the correct granular input feature. In this way, for example, the interface may utilize a higher confidence level of region-level input to improve the interface tolerance for low-accuracy selections of processing granular input features.

[0049] In a text entry example, a keyboard configured according to example aspects of the present disclosure may have a first set of letter keys associated with a left region of the keyboard and a second set of letter keys associated with a right region of the keyboard. Because the size of the key is related to the size of the left and right regions, the spatial accuracy associated with selecting a specific key associated with a desired symbol may be much higher than the spatial accuracy associated with selecting the correct region associated with the desired symbol. In this way, for example, a sequence of left-right-left-left may be received with a higher confidence than a particular sequence of received keystrokes. The higher confidence sequence may then be used to predict the desired sequence of symbols, thereby improving the keyboard's robustness to low-accuracy key input.

[0050] Input interfaces according to example aspects of the present disclosure may provide multiple benefits over conventional input processing techniques, such as by improving accessibility to computing device functionality for users of varying abilities. For example, in contrast to the currently described techniques, conventional interfaces for inputting text into a computer system may typically be implemented using a digitized variation of a traditional (i.e., physical) keyboard layout (e.g., a QWERTY layout for the English language or a QWERTZ layout for the German language) in which each character has a single key. These conventional interfaces assume that the operating user has sufficient flexibility on his or her part to interact with the individual character keys in order to input text. However, users of varying abilities (e.g., whose abilities are permanently or temporarily determined by physical abilities, environmental constraints, clothing constraints, etc.) may not have the requisite degree of flexibility to input text on a conventional system, such that a conventional interface would make such a conventional system difficult to use (if possible at all). In addition, existing low-accuracy input formats (e.g., Morse code) typically require users to learn an entirely new input communication paradigm.

[0051] Additionally, the example interfaces described herein may facilitate interaction with and control of a computing device that has constraints for any or all users. Such constraints may be inherent. For example, some computing devices have small input interfaces, so that granular input may be impractical (e.g., the touch screen of a smart watch is small relative to a finger or stylus, and the physical space for buttons is limited, etc.). Some computing devices may have constraints determined by convention or convenience. For example, televisions are traditionally not associated with full keyboards for text entry, but are traditionally only equipped with remote controls with limited alphanumeric input options. Due to the small input interface, the remote control has about a dozen small keys, a non-QWERTY layout, and the need to press buttons multiple times to select letters, the alphanumeric input option may be difficult to use. Similarly, game consoles tend to be used primarily with game input controllers, so that switching to different input devices (e.g., full keyboards) may be inefficient and inconvenient.

[0052] In some embodiments, the user interface may include a typing system with a multi-region keyboard interface (e.g., a virtual keyboard, an on-screen keyboard, a physical keyboard) that can be used to operate a computing device. (The various examples described herein are presented in the context of symbol or other text entry. However, it should be understood that the scope of the present disclosure is not limited to a text entry device.) The typing system can operate a desktop computer, a laptop computer, a tablet computer, a smartphone, a wearable computing device, a virtual reality system, a smart TV, a game console, or virtually any computing device. Additionally or alternatively, the typing system can enable a user to operate a computer device with the aid of a peripheral device such as an assistive technology interface (e.g., a head mouse, a switch control, a gesture detection device, a gaze detection device, an electromyography sensor).

[0053] In some implementations, a graphical keyboard interface may be utilized with a computing device such as a smartphone and enable operations for users whose context, scenario, or permanently limited flexibility would otherwise prevent them from using a standard keyboard to enter text. For example, typing on a small touchscreen-based mobile phone keyboard may be more challenging when standing on a bumpy train car than when sitting in a cafe. Additionally, the techniques described herein may unlock a significantly improved text entry experience on computing devices that may have been originally or traditionally unsuitable for the task due to their size or interface constraints. For example, a smartwatch may have a screen that is too small for a traditional keyboard that requires direct pressing, and there are only cumbersome text entry options for the screen. In another example, a virtual reality headset may have a virtual keypad for which a handheld pointer or controller is not suitable due to lack of accuracy. In yet another example, a television and a game console may have a remote control that typically employs a directional navigation control that requires multiple iterative clicks across the keyboard to enter text. In yet another example, an in-car navigation system may require the use of a large screen in the interior design to facilitate information entry.

[0054] In some implementations, word and / or phrase prediction technology can assist in text entry in a computing system. By predicting the word or phrase that a user intends to enter, the word and / or phrase prediction technology can enable a reduction in the number of keystrokes required. Additionally, the prediction technology can suggest words after the word currently being entered. Additionally, the prediction technology can include other features such as spell checking, speech synthesis, and shortcuts for frequently used words.

[0055] Example aspects of the present disclosure relate to a data entry system for a computing device that uses predictive text modeling techniques to implement a minimal interface optimized for robustness in handling low-accuracy input (e.g., for user operation in a scenario with limited flexibility). A typing system (e.g., a system for data entry) may include a multi-region (e.g., two-button, three-button, four-button) keyboard whose characters (e.g., letters, punctuation marks) are grouped into different regions. Symbols may be grouped into multiple regions, each of which may be presented as a graphical button or a collection of buttons in a multi-region graphical keyboard interface. For example, a graphical button may be larger than a single key button to achieve a large surface, thereby facilitating user interaction. The typing system may also include a suggestion area (e.g., a prediction area) in which directly matched and / or predicted text entries are displayed. The suggestion area may be graphically larger than a single key button to facilitate user interaction. Additionally, the typing system may include a sequence area that presents a sequence of previously entered user inputs to the user (e.g., a sequence of selected graphical buttons).

[0056] The techniques described herein may provide a number of technical effects and benefits. For example, as previously indicated, the techniques described herein may render a multi-region keyboard interface easier to operate (particularly for users with flexibility constraints). Additionally, the predictive techniques described herein predict and suggest related words and / or phrases, which may allow a user to enter data faster, thereby reducing the time spent typing, editing, and other graphical user interface interactions, thereby saving computing resources (e.g., energy, processing cycles, network bandwidth, etc.). One or more aspects of the graphical keyboard interface may be configured to provide access to an easy-to-use keyboard interface for users with limited flexibility.

[0057] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

[0058] Example Apparatus and Systems

[0059] Figure 1A A block diagram of an example computing system 100 that performs assisted and predictive input techniques according to an example embodiment of the present disclosure is depicted. System 100 may include a user computing device 102, a server computing system 130, and a training computing system 150 that may be communicatively coupled via a network 180.

[0060] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, a smart device (e.g., a smart TV, smart appliances), a virtual reality system, an augmented reality system, or any other type of computing device.

[0061] The user computing device 102 may include one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 executed by the processor 112 to cause the user computing device 102 to perform operations.

[0062] In some implementations, the user computing device 102 may store or include one or more prediction models 120. For example, the prediction model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model). Figure 2 An example prediction model 120 is discussed further in .

[0063] In some implementations, one or more prediction models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single prediction model 120.

[0064] Additionally or alternatively, one or more prediction models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the prediction models 140 may be implemented as part of a web service by the server computing system 140. Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0065] The user computing device 102 may also include one or more user input components 122 (e.g., a graphical keyboard 124, a microphone 126, a camera 128, an optical sensor, a touch sensor, a button, a switch, etc.) that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). A touch-sensitive component may be used to implement the graphical keyboard 124. Other example user input components include a microphone 126, a graphical keyboard 124, a camera 128, or other devices by which a user may provide user input.

[0066] The user computing device 102 may also include a graphical keyboard 124 that receives user input (e.g., for use via a touch screen). The graphical keyboard 124 may be provided to one or more user devices (e.g., a computer, a smartphone, a tablet computing device, a wearable computing device) (e.g., as part of an operating system (OS), a third-party application, a plug-in). One or more aspects of the graphical keyboard 124 may be configured to provide suggested words and / or phrases using the prediction model 120 and / or the prediction model 140. The graphical keyboard 124 may be a multi-region graphical keyboard interface having multiple key regions. The multiple key regions may include a first key region, a second key region, a third key region, and a fourth key region. The first key region may have a first set of keys, the second key region may have a second set of keys, and so on.

[0067] According to aspects of the present disclosure, the graphical keyboard 124 may include a graphical keyboard interface (e.g., for use via a touch screen) that may be provided to or by the user computing device 102 (e.g., as part of an operating system (OS), a third-party application, a plug-in). Figure 3B , the first key area 311, the second key area 312, the suggestion area 313, and the sequence area 316 may be associated with such a graphical keyboard interface. The dynamic keyboard interface may be provided in association with one or more applications executed by the user computing device 102. For example, the graphical keyboard 124 may be associated with an application (e.g., Figure 1B The email application 11, the virtual keyboard application 12, the text message application 13, etc. in the example are associated with the virtual keyboard application 12, and the text message application 13 is ... FIG. 3A to FIG. 3H As illustrated, a graphical keyboard interface may be provided in association with such an application.

[0068] The server computing system 130 may include one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 executed by the processor 132 to cause the server computing system 130 to perform operations.

[0069] In some implementations, server computing system 130 may include or otherwise be implemented by one or more server computing devices. Where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0070] As described above, the server computing system 130 may store or otherwise include one or more prediction models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a Transformer model). Figure 2 An example prediction model 140 is further discussed in .

[0071] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150, which may be communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0072] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 154 may store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0073] The training computing system 150 may include a model trainer 160 that trains the prediction models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques such as, for example, error back propagation. For example, a loss function may update one or more parameters of the model (e.g., based on the gradient of the loss function) through model back propagation. Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations.

[0074] In some implementations, performing error back-propagation may include performing truncated back-propagation through time.The model trainer 160 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0075] Specifically, model trainer 160 may train prediction models 120 and / or 140 based on a collection of training data 162. Training data 162 may include, for example, previous user interactions with the suggestion region. Training data 162 may include selection rates associated with user selections of words or phrases presented in the suggestion region of the graphical user interface.

[0076] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0077] The model trainer 160 may include computer logic for providing the desired functionality. The model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 may include a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 may include a set of one or more computer executable instructions stored in a tangible computer readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0078] In some implementations, the prediction models 120 and / or 140 may be trained by the model trainer 160 using federated learning techniques to allow for user-specific or device-specific model training and global updates.

[0079] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0080] The prediction models 120 and / or 140 (e.g., machine learning models) described in this specification can be used for various tasks, applications, and / or use cases. In some implementations, the prediction models 120 and / or 140 may include a Transformer model. A Transformer model can be a neural network that learns context and therefore meaning by tracking relationships in sequence data (e.g., letters in words, words in sentences). Additionally, the prediction models 120 and / or 140 may include a recurrent neural network (RNN), such as a long short-term memory (LSTM) network. Additionally, the prediction models 120 and / or 140 may include an autoregressive model and / or a feedforward network that can be distilled from any model.

[0081] In some implementations, when the prediction model 120 and / or 140 uses a Transformer model, the prediction model 120 and / or 140 may also utilize masking techniques by using an encoder and a decoder.

[0082] In some implementations, when the prediction models 120 and / or 140 may use a beam search algorithm. The beam search algorithm may be a heuristic search algorithm that explores a graph by expanding the most promising nodes in a finite set. The beam search may be an optimization of a best-first search (e.g., defining a set of suggested words and / or phrases) that enables reduced memory requirements.

[0083] In some implementations, the input to the prediction model of the present disclosure may be text or natural language data. The prediction model may process text or natural language data to generate an output. As an example, the prediction model may process natural language data to generate a language encoding output. As another example, the prediction model may process text or natural language data to generate a potential text embedding output. As another example, the prediction model may process text or natural language data to generate a translation output. As another example, the prediction model may process text or natural language data to generate a classification output. As another example, the prediction model may process text or natural language data to generate a text segmentation output. As another example, the prediction model may process text or natural language data to generate a semantic intent output. As another example, the prediction model may process text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data with higher quality than the input text or natural language, etc.). As another example, the prediction model may process text or natural language data to generate a prediction output.

[0084] In some implementations, the input to the prediction model of the present disclosure may be image data. The prediction model may process the image data to generate an output. As an example, the prediction model may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, encoded representation of the image data, hashing of the image data, etc.). As another example, the prediction model may process the image data to generate an image classification output. As another example, the prediction model may process the image data to generate a prediction output.

[0085] In some implementations, the input to the prediction model of the present disclosure may be speech data. The prediction model may process speech data to generate an output. As an example, the prediction model may process speech data to generate a speech recognition output. As another example, the prediction model may process speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of speech data, etc.). As another example, the prediction model may process speech data to generate a text representation output (e.g., a text representation of input speech data, etc.). As another example, the prediction model may process speech data to generate a predicted output. In some cases, the input includes audio data representing a spoken utterance, and the task is a speech recognition task. The output may include a text output mapped to the spoken utterance.

[0086] In some implementations, the input to the prediction model of the present disclosure can be encoded data (e.g., a sequence representation of keyboard button inputs, such as areas of an input interface associated with finer-grained inputs, etc.). The prediction model can process the encoded data to generate an output. As an example, the prediction model can process the encoded data to generate words and / or phrases. As another example, the prediction model can process the encoded data to generate a predicted output. For example, in some embodiments, the encoded data can be input into a language model to predict one or more natural language outputs associated with the encoded data.

[0087] In some implementations, the input to the predictive model of the present disclosure may be sensor data (e.g., hand and / or finger gestures captured from a camera of a computing device). The predictive model may process the sensor data to generate an output. As an example, the predictive model may process the sensor data to generate words and / or phrases. As another example, the predictive model may process the sensor data to generate a prediction output. As another example, a machine learning model may process the sensor data to generate a detection output. For example, the predictive model may process the sensor data to confirm the selection of a word or phrase presented in a suggestion area of ​​a graphical user interface. As another example, the predictive model may process the sensor data to generate a visualization output.

[0088] For example, the graphical keyboard interface may receive a user's facial gesture as input. Facial gestures may indicate a specific task to be performed by the computing system. In some cases, the user may control the mapping action on the display of the computing system by moving the user's eyes (e.g., left and right, looking up and down, winking, blinking, or other facial gestures). In some cases, the graphical keyboard interface may be custom programmed by a developer or user to perform certain tasks based on gestures (e.g., facial gestures, hand gestures, arm gestures, foot gestures). The optical sensor (e.g., camera) of the computing device may capture the gesture. For example, a user with a dexterity disorder may program the graphical keyboard to perform a task (e.g., accepting a first suggested word, moving the selection of the suggested word to the right, left, up, or down) based on gestures that the user may perform. In some cases, the developer may pre-program the graphical keyboard interface to perform tasks based on gestures using pre-assigned gesture mappings. The gesture mapping may depend on the type of dexterity disorder that the user may suffer from. The user may select the dexterity disorder type when starting the graphical keyboard interface or by changing the settings of the graphical keyboard interface. The graphical keyboard interface may be programmed based on the selection of the dexterity impairment type to perform specific actions based on different postures.

[0089] Figure 1AAn example computing system that can be used to implement the present disclosure is illustrated. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 may be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0090] Figure 1B Depicted is a block diagram of an example computing device 10 performing in accordance with an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0091] The computing device 10 includes a plurality of applications (e.g., application 1 to application N). Each application contains its own machine learning library and prediction model (e.g., prediction model 120, prediction model 140). For example, each prediction model may include a machine learning model. Example applications include an email application 11, a virtual keyboard application 12, a text messaging application 13, a dictation application, a browser application, etc.

[0092] like Figure 1B As illustrated, each application can communicate with a plurality of other components of the computing device, such as, for example, one or more sensors 21 (e.g., buttons, cameras), a graphical user interface 22 (e.g., a virtual keyboard), an audio input 23 (e.g., a microphone), a scene manager, a device state component, and / or additional components 24. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0093] Figure 1C Depicted is a block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0094] The computing device 50 includes multiple applications (e.g., an email application, a virtual keyboard application, a text messaging application, a web browsing application). Each application can communicate with the central intelligence layer 55. Example applications include a text messaging application, an email application 51, a dictation application, a virtual keyboard application 52, a text messaging application 53, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer 55 (and the predictive models stored therein) using an API (e.g., a common API across all applications).

[0095] The central intelligence layer 55 may include multiple predictive models (e.g., machine learning models). Figure 1CAs illustrated, a corresponding prediction model may be provided for each application, and these corresponding prediction models may be managed by the central intelligence layer 55. In other implementations, two or more applications (e.g., email application 51 and text messaging application 53) may share a single prediction model. For example, in some implementations, the central intelligence layer 55 may provide a single prediction model for all applications. In some implementations, the central intelligence layer 55 may be included in or otherwise implemented by the operating system of the computing device 50.

[0096] The central intelligence layer 55 may communicate with a central device data layer 60. The central device data layer 60 may be a centralized data repository for the computing device 50. Figure 1C As illustrated, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors 71, a graphical user interface 72 (e.g., a virtual keyboard), an audio input 72, a scene manager, a device state component, and / or additional components. In some implementations, the central device data layer 55 can communicate with each device component using an API (e.g., a private API).

[0097] Example Prediction Model Arrangement

[0098] Figure 2 A block diagram of an example prediction system 200 is depicted in accordance with an example embodiment of the present disclosure. In some implementations, a prediction model 204 is trained to receive a set of input data 202 and determine (e.g., provide, output) suggested words and / or phrases 206. The input data 202 may describe user input obtained from a multi-region graphical keyboard interface. As a result of obtaining the input data 202, the prediction model 204 may determine the suggested words and / or phrases 206. The suggested words and / or phrases 206 may be presented on a suggestion area of ​​the graphical user interface. Thus, in some implementations, the prediction system 200 may include a prediction model 202 operable to predict words and / or phrases. The prediction model may be, for example, Figures 1A to 1C The machine learning model described.

[0099] In some implementations, the computing system (user computing device 102, server computing device 130, training computing device 150, computing device 10, computing device 50) may use Figure 2 The example prediction system 200 described in is used to process input data 202 to determine one or more suggested words and / or phrases 206.

[0100] The computing system may access (e.g., obtain, receive) input data 202 from the multi-region graphical keyboard interface. For example, a user may use the graphical keyboard 124 to input a first input, a second input, a third input, and the like. The keys of the graphical keyboard 124 may be grouped in a plurality of key regions. Each input may correspond to a user selecting a region from a plurality of key regions. In some implementations, each input may correspond to a user selecting any key in a region of a plurality of key regions. For example, a user may select a key or region by touching a region presenting a key or region in a graphical user interface. In another example, a user may select a region by pushing a button corresponding to a region in a plurality of regions (e.g., a button on a smart watch, a button on a remote control). In yet another example, when using a virtual reality system, a user may select a key or region by gazing at a region associated with a key or region and using a gesture (e.g., a wink, a hand gesture, pushing a button on a handheld device). In yet another example, a hand or arm gesture associated with a region in a plurality of key regions may be used to select the region.

[0101] In some cases, the prediction model 204 can dynamically determine suggested words and / or phrases based on previous user interactions (e.g., user typing history, typing history of a subset of users) and / or based on the current context of a conversation with the user. Additionally, the prediction model 204 can predict the next key to be selected or the action to be performed by the graphical keyboard interface. The predicted key or predicted action can be emphasized (e.g., highlighted) on the graphical user interface.

[0102] Additionally, the prediction model 204 may be trained by the model trainer 208 using various training or learning techniques such as, for example, error back propagation. For example, a loss function may be used to update one or more parameters of the model through model back propagation (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations. Figure 1A Model trainer 160 in can be an example of model trainer 208 in prediction system 200.

[0103] In some implementations, model trainer 208 may train prediction model 204 based on a set of training data 209. Training data 209 may include Figure 1A 209. Additionally, training data 209 may include, for example, user feedback 210. User feedback 210 may include previous user interactions with suggested words and / or phrases 206. User feedback 210 may include selection rates associated with user selections of words or phrases presented in a suggestion area of ​​a graphical user interface.

[0104] For example, in some embodiments, the prediction model 204 may include a machine learning model configured to receive an input sequence and generate an output sequence. For example, the prediction model 204 may include a Transformer architecture (e.g., an encoder structure, a decoder structure, one or more self-attention heads, etc.). For example, the prediction model 204 may include a model configured to operate on one or more input embeddings to generate a natural language output.

[0105] For example, a pattern or sequence of selected regions may provide an input encoding. For example, the input encoding may include a region sequence encoding indicating a sequence of selections of one or more regions. For example, the received input sequence may include: a selection of a first region, a selection of a second region, a selection of a first region, and a selection of a first region (e.g., first-second-first-first). In this example, for example, the region sequence encoding may include an embedding indicating a pattern first-second-first-first (e.g., "FSFF", "1211", a digital embedding thereof, etc.). The region sequence embedding may be processed by the prediction model 204 to generate one or more suggested outputs (e.g., a probability distribution of suggested outputs) that may correspond to the region sequence embedding. For example, in this way, higher confidence region-level inputs may be used to predict granular-level outputs with improved robustness for processing low accuracy inputs.

[0106] In some embodiments, the prediction model 204 may be trained based on a training data set containing a natural language corpus. Training may be performed in an unsupervised manner by: for a given natural language sequence (e.g., a word), a corresponding region (e.g., a keyboard region) sequence may be automatically constructed based on the position of the symbol input of the language sequence in the region. Using the corresponding sequence (or its embedding) as input, the prediction model 204 may be trained to predict a given natural language sequence. For example, in this way, the prediction model 204 may be trained in an unsupervised manner. Such training may be performed in addition to personalized training based on user interaction with the suggestion region (e.g., by selecting one or more suggested inputs as desired inputs).

[0107] Example GUI

[0108] FIG. 3A to FIG. 3H An example graphical user interface (GUI) is depicted according to an example embodiment of the present disclosure.

[0109] refer to Figure 3A, the GUI (e.g., graphical keyboard 300) may include a first set of keys in a first key area 301 and a second set of keys in a second key area 302. Additionally, graphical keyboard 300 may include a suggestion area 303 having a plurality of words 304. In some implementations, the first set of keys may have a plurality of keys (e.g., 20 keys, 15 keys, 10 keys, 8 keys), and the second set of keys may have a plurality of keys.

[0110] refer to Figure 3B , the graphical keyboard 310 may include a first set of keys in a first key area 311 and a second set of keys in a second key area 312. Additionally, the graphical keyboard 300 may include a suggestion area 313 having a plurality of words 314 and a plurality of phrases 315. Additionally, the graphical keyboard 310 may include a sequence area 316 (also referred to herein as a breadcrumb user interface) having a sequence of symbols associated with each selection area from a previous user input. In some cases, each key area in the plurality of key areas may correspond to a unique symbol. In this example, the first key area 311 may be associated with a first symbol (e.g., a transparent dash symbol), and the second key area 312 may be associated with a second symbol (e.g., a shaded dash symbol). As depicted in the sequence area 316, the first sequence element 317 is the second symbol, the second sequence element 318 is the first symbol, and the third sequence element 319 is the second symbol. Thus, the sequence area indicates that a first user input includes keys from the second key area 312 by displaying the second symbol, and then indicates that a subsequent second user input includes keys from the first key area 311 by next displaying the first symbol, and so on. As the user continues to create these sequences, breadcrumbs may be presented in the sequence area 316 as a reminder of previous inputs.

[0111] In some implementations, the graphical keyboard 300 may have a third key region that may be associated with a third key set. The third key region may also be associated with a third symbol to be presented in the sequence region 316. In some implementations, the graphical keyboard 300 may have a fourth key region that may be associated with a fourth key set. The fourth key region may also be associated with a fourth symbol to be presented in the sequence region 316.

[0112] In some implementations, the key sets in the first key area, the second key area, and the other key areas (e.g., the third key area, the fourth key area) may be minimized on the graphical user interface. For example, once the user acquires touch type memory (e.g., remembers which keys are in which key areas), the user may not need to view different keys in different key areas. With this implementation, areas of the graphical user interface can be used for other purposes, such as presenting only the suggestion area and / or the sequence area. For example, in one embodiment, only the suggestion area may be presented on the graphical user interface of a computing device (e.g., a smart watch), without presenting the first key area and the second key area. In another embodiment, only the suggestion area and the sequence area may be presented on the graphical user interface of the computing device.

[0113] In some embodiments, the regions can be associated with inputs that are not on the graphical user interface. For example, the regions can be associated with corresponding physical buttons or other inputs. For example, the regions can be associated with buttons on the side of a smartwatch, a TV remote control, a game controller, etc., such that the graphical user interface can display suggested regions.

[0114] Figure 3B A graphical keyboard 310 that may be presented on a smartphone according to an example embodiment is depicted. The graphical keyboard 310 may include an input area (e.g., for composing a message) and a send element that may be used to transmit data located in such an input area to an application (e.g., Figure 1B 10) in the email application 10). The positions of the first key area 311, the second key area 312, the suggestion area 313, and the sequence area 316 on the GUI can be modified based on user requests or design choices of developers. In this example, as depicted in the sequence area, the user entered a sequence of right button-left button-right button-right button-right button (i.e., RLRRR or shaded hyphen / white hyphen / shaded hyphen / shaded hyphen / shaded hyphen), based on which the system can predict the suggested words "Hello", "Kelly", "People", etc. The user can select one of the suggested words 314 or suggested phrases 315 presented on the suggestion area 313.

[0115] Reference now FIG. 3C to FIG. 3E , according to an example embodiment of the present disclosure, the graphical keyboards 320 , 330 , 340 depict an example of a typing process. Figure 3C The graphical keyboard 320 in FIG. 3 is a two-button keyboard having a first set of keys in a first key area 321 and a second set of keys in a second key area 322. As depicted in this example, the first key area 321 may have more characters (e.g., letters) than the second key area 322, which may be a design choice of the developer of the graphical keyboard 320.

[0116] When the user continues Figure 3D , sequence area 331 and suggestion area 332 may be updated after each user input. As depicted in sequence area 331, the last four user inputs are RRRL (i.e., shaded dash / shaded dash / shaded dash / white dash), and a predictive model (e.g., predictive model 204) may determine suggested words and phrases to present in suggestion area 332 of graphical keyboard 330.

[0117] refer to Figure 3E , the user may select the desired word 'long' from suggestion area 342 of graphical keyboard 340. Based on the word selection, the words and / or phrases in suggestion area 342 are updated by using a prediction model (eg, prediction model 204).

[0118] The prediction model may determine suggested words based on the machine learning techniques described herein. In this example, the suggested words may be ranked and presented based on the ranking of each word. In some cases, words that fully match the input sequence (e.g., for input RLRR, words in the prediction dictionary that match the input pattern of RLRR) may be ranked highest and may be presented at the top of the suggestion area. When two words have the same ranking, the more frequently used word may be ranked higher. Additionally, partial matches of the input sequence (e.g., for input RLRR, ranked matches of longer words that begin with RLRR) may be presented in the suggestion area after fully matched words. In addition, phrase matches of the input sequence (e.g., for input RLRR, ranked matches of phrases that begin with RLRR) may also be presented in the suggestion area. In some implementations, the highest ranked words and / or phrases may be presented at the top of the suggestion area.

[0119] In some implementations, word and phrase suggestions can be paged when the top predictions (e.g., suggestions) do not match the user's intended input. For example, the area of ​​a keyboard or other input device can be used to generate signals to the predictive model, rather than just the explicit or direct source of text input. Additionally, the keyboard can maintain the familiarity of an existing keyboard layout so that the user does not have to learn a complex new text encoding scheme.

[0120] refer to Figure 3F , the graphical keyboard may be implemented on different types of computing systems (such as a smart watch 350). Additionally, the computing system may include a first button 351, a second button 352, and the like. In some implementations, a user may select a key area from a plurality of key areas by pressing the first button 351 or the second button 352. In some implementations, when the graphical keyboard includes a third key area (e.g., Fig. 3I), the user may select the third area by pressing a third button (not shown). In some implementations, when the graphical keyboard includes a fourth key area (e.g., Figure 3J ), the user may select the fourth area by pressing a fourth button (not shown). Additionally or optionally, the user may select a key area from a plurality of key areas by touching (e.g., pressing) an area on the dial where the corresponding key area is presented. For example, the user may touch the left side of the dial to select the first key area, or the user may touch the right side of the touch screen (e.g., the dial) to select the second key area. In addition, the graphical keyboard may include a suggestion area (not shown) where suggested words and / or phrases are presented. The user may select a suggested word or phrase by pressing an area on the touch screen (e.g., the dial) where the suggested word or phrase is presented. In addition, in some cases, the user may select a highlighted suggested word or phrase by pressing a button (e.g., the first button 351, the second button 352), by pressing a combination of buttons (e.g., the first button 351 and the second button 352), or by a hand gesture.

[0121] In some implementations, the graphical keyboard may be presented on a wearable computing device (e.g., a smart watch) that can be worn, for example, on the arm of a user. The wearable computing device may include a housing that defines a cavity. Additionally, the wearable computing device may include a button that is partially positioned in a recess defined by an outer surface of the housing. The button may be positioned at the periphery (e.g., left side, right side, edge) of the housing. In addition, the button may include a plurality of sensors (e.g., strain sensors, ultrasonic sensors, motion sensors, optical sensors, and force sensors) that are configured to detect actuation of the button via input provided by a user, and whether the user is touching the button. The wearable computing device may include one or more processors positioned in the cavity. The wearable computing device may include a printed circuit board disposed in the cavity. The computing device 100 may further include a battery (not shown) disposed in the cavity. Additionally, the computing device may include a motion sensor positioned in the cavity of the housing. For example, the motion sensor may also include an accelerometer that may be used to capture motion data indicating motion of the wearable computing device. Alternatively or additionally, the motion sensor may also include a gyroscope, which may also be used to capture motion information about the wearable computing device. The hand or arm gesture may be determined based on the motion data obtained from the motion sensor. The wearable computing device may include a display screen to present a plurality of key areas of a graphical keyboard. As previously discussed, the user input may include pressing (e.g., touching) an area on the display screen associated with a key area in the plurality of key areas.

[0122] refer to Figure 3G, a graphical keyboard may be implemented on the smart TV 360. In some implementations, the computing system may include a remote controller 361 having a plurality of buttons. For example, a plurality of key regions may correspond to different buttons on the remote controller 361. A user may select a key region from the plurality of key regions by the user pressing a button on the remote controller 361.

[0123] refer to Figure 3H , a graphical keyboard may be implemented on a tablet computer 370. In some implementations, the first key region 371 may be separated from the second key region 372 by a predetermined distance to make it easier for the user to operate the GUI. For example, by placing the key region closer to the edge of the tablet computer 370, it may be easier for the user to type when holding the tablet computer with both hands.

[0124] refer to Fig. 3I , the graphical keyboard 380 can be implemented using a three-region keyboard. The graphical keyboard 380 can have a first key region 381, a second key region 382, ​​and a third key region 383. Additionally, the graphical keyboard 380 can include a sequence region 384 that presents a first symbol associated with the selection of the first key region, a second symbol associated with the selection of the second key region, and a third symbol associated with the selection of the third key region. In addition, the graphical keyboard 380 can include a suggestion region 385 that has suggested words and / or phrases based on the selection of the key regions.

[0125] refer to Figure 3J , the graphical keyboard 390 can be implemented using a four-region keyboard. The graphical keyboard 390 can have a first key region 391, a second key region 392, a third key region 393, and a fourth key region 394. Additionally, the graphical keyboard 390 can include a sequence region 395 that presents a first symbol associated with the selection of the first key region, a second symbol associated with the selection of the second key region, a third symbol associated with the selection of the third key region, and a fourth symbol associated with the selection of the fourth key region. In addition, the graphical keyboard 390 can include a suggestion region 396 that has suggested words and / or phrases based on the selection of the key regions.

[0126] Example Method

[0127] Figure 4 A flow chart depicts an example method of performing predictive and assistive techniques using a multi-region graphical keyboard interface according to an example embodiment of the present disclosure. Figure 4 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The various steps of method 400 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0128] At 402, a system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) may present a graphical keyboard having a plurality of key regions on a display of the computing device. The plurality of key regions include a first key region having a first key set and a second key region having a second key set. For example, the first key set may be a plurality of keys in a standard keyboard. The second key set may be a plurality of keys in a standard keyboard that are different from the first key set.

[0129] In some implementations, the first key set may have a plurality of keys (e.g., 20 keys, 15 keys, 10 keys, 8 keys), and the second key set may have a plurality of keys. Additionally, only one user input may select a key in the first key set. For example, a user may select the key by pressing (e.g., touching an area associated with the first key set in a graphical user interface, pressing a physical button on a smart watch or remote control) the area associated with the first key set only once. Prediction models 120 and / or 140 may determine the key from the first key set based on a sequence of inputs received from the user.

[0130] In some implementations, the graphical keyboard can include a QWERTY layout for the English language, or a QWERTZ layout for the German language.

[0131] In some implementations, the first set of keys can be keys located in a left column of the graphical keyboard, and the second set of keys can be keys located in a right column of the graphical keyboard.

[0132] In some implementations, the first key region has a larger number of keys than the second key region. For example, based on design research, having different numbers of keys in each key region can make the graphical keyboard interface easier to access and easier to operate.

[0133] In some implementations, the plurality of key regions may include a third key region having a third set of keys. The third set of keys is different from the first set of keys and the second set of keys.

[0134] In some implementations, the plurality of key regions include a fourth key region having a fourth key set. For example, the first key set may be keys located in the upper left quadrant of the graphical keyboard, the second key set may be keys located in the upper right quadrant of the graphical keyboard, the third key set may be keys located in the lower left quadrant of the graphical keyboard, and the fourth key set may be keys located in the lower right quadrant of the graphical keyboard.

[0135] At 404, the system may receive a first input selecting a first selection area from a plurality of key areas. For example, the user may select the first selection area by touching an area on the graphical keyboard that is in the first key area or the second key area.

[0136] In some implementations, the computer device includes a first button and a second button, and the first input may be received by a user pressing the first button or the second button.

[0137] At 406, the system can determine a first suggestion and a second suggestion based at least in part on the first input. For example, the first suggestion and the second suggestion can be determined using a predictive model 204 using techniques described herein.

[0138] In some implementations, the first suggestion can be a word and the second suggestion can be a phrase.

[0139] At 408, in response to the first input, the system may present an updated graphical keyboard having a plurality of key regions and a suggestion region on a display of the computing device. The suggestion region may include a first suggestion and a second suggestion. The first suggestion or the second suggestion may be selected by user input. Once a suggestion is selected, the selected suggestion may be presented in a different region of the graphical keyboard interface.

[0140] In some implementations, the system can receive the gesture via a sensor coupled to the computing device. Additionally, in response to the gesture, the system can select the first suggestion.

[0141] Figure 5 A flow chart depicting an example method of updating suggested words and / or phrases in a suggestion area of ​​a graphical keyboard interface according to an example embodiment of the present disclosure. Figure 5 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The various steps of method 500 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0142] At 502, continuing with method 400, the system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) may receive a second input selecting a second selection area from a plurality of key areas. For example, after operation 404 in method 400, the system may receive the second input.

[0143] At 504, the system may rank a plurality of words associated with the first input and the second input based at least in part on the first input and the second input. For example, a prediction model (e.g., prediction model 204) may rank a plurality of words based on the first input and the second input.

[0144] In some implementations, the plurality of words are further ranked based on previous user interactions with the plurality of words. In some implementations, the plurality of words are also ranked based on previous user interactions with the suggestion area. For example, words that are frequently selected by a user may be ranked higher than words that are not selected as frequently by the user.

[0145] At 506, the system can determine updated suggestions from the plurality of words based on the rankings of the plurality of words.As previously described, a predictive model (eg, predictive model 204) can determine updated suggestions.

[0146] At 508, the system can present the updated suggestions in a suggestion area of ​​the updated graphical keyboard.

[0147] In some implementations, the updated suggestion can be a suggested word that exactly matches the first input and the second input.

[0148] In some implementations, the updated suggestion is a suggested word that partially matches the first input and the second input. For example, the suggested word can begin with a character (e.g., a letter) associated with the first input and the second input.

[0149] In some implementations, the updated suggestion can be a suggested phrase that partially matches the first input and the second input, and where the suggested phrase begins with characters (eg, letters) associated with the first input and the second input.

[0150] Figure 6 A flow chart of an example method of presenting symbols in a sequence area of ​​a graphical keyboard interface according to an example embodiment of the present disclosure is depicted. Figure 6 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The various steps of method 600 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0151] In some implementations, the graphical keyboard interface can include a sequence area.

[0152] At 602 , the system (eg, user computing system 102 , server computing device 130 , computing device 10 , computing device 50 ) may create a sequence area in an updated graphical keyboard. FIG. 3A to FIG. 3I Examples of suggested regions are illustrated in .

[0153] At 604, the system may determine a first determined symbol from a plurality of symbols based on the first selection area. The plurality of symbols may include a first symbol corresponding to the first key area (e.g., a shaded connector) and a second symbol corresponding to the second key area (e.g., a white connector). In one example, the first selection area may be the first key area.

[0154] At 606, the system can receive a second input selecting a second key region.

[0155] At 608, the system may determine a second definite symbol from the plurality of symbols based on the second input. In this example, the second definite symbol may be different from the first definite symbol because the second selection area may be a second key area.

[0156] At 610 , the system can present the first determination symbol and the second determination symbol in a sequence area of ​​the updated graphical keyboard.

[0157] In some implementations, the first definite symbol can be a first shape having a first color (eg, a shaded connector), and the second definite symbol can be the first shape having a second color (eg, a white connector).

[0158] Figure 7 A flowchart depicting another example method of performing predictive and assistive techniques using a graphical keyboard interface according to an example embodiment of the present disclosure. Figure 7 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The steps of method 700 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0159] At 702, a system (e.g., user computing system 102, server computing device 130, computing device 10, computing device 50) may render a graphical array of input features on a display component, the graphical array of input features having: a first plurality of input features associated with a first region; and a second plurality of input features associated with a second region.

[0160] In some implementations, the input features can correspond to language symbols.

[0161] In some implementations, the first area can correspond to a first region of a graphical keyboard, and the second area can correspond to a second region of the graphical keyboard.

[0162] At 704, the system may determine a region sequence code based on one or more inputs received from the input component, the region sequence code describing a sequence including one or more selections of the first region or the second region. Figure 2The input data 202 in may be an example of one or more inputs received from an input component.

[0163] In some implementations, the one or more inputs may include inputs associated with input signals assigned to the first region and the second region, respectively. The input signals may correspond to one or more physical toggle switches. Additionally or alternatively, the input signals may correspond to areas on the touch screen that cover the first region and the second region, respectively. Additionally or alternatively, the input signals may correspond to peripheral components, respectively.

[0164] At 706, the system can generate one or more suggested inputs based on the region sequence encoding. For example, FIG. 3A to FIG. 3H The graphical user interface depicted in exemplifies one or more suggested inputs generated at 706 .

[0165] In some implementations, one or more suggested inputs may include suggested words for a language (e.g., Figure 2 Suggested words and / or suggested phrases generated by the predictive model 204 in FIG.

[0166] In some implementations, the one or more suggested inputs generated at 706 may be generated by inputting the region sequence encoding into a prediction model (e.g., prediction model 204, a machine learning model), and using the machine learning model to generate the one or more suggested inputs.

[0167] In some implementations, the one or more suggested inputs generated at 706 can be generated by generating a probability distribution corresponding to the one or more suggested inputs.

[0168] In some implementations, the prediction model (e.g., prediction model 204, machine learning model in method 700) may include a natural language model. Additionally or alternatively, the prediction model (e.g., machine learning model) may include one or more Transformer architectures.

[0169] In some implementations, a predictive model (e.g., a machine learning model) can be trained on an unsupervised dataset for a target keyboard layout based on region sequences associated with input of symbols corresponding to words in a vocabulary.

[0170] Additional disclosure

[0171] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and partitions of tasks and functionality between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0172] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, may easily produce changes, modifications, and equivalents of such embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature illustrated or described as part of an embodiment may be used together with another embodiment to produce a further embodiment. Therefore, the present disclosure is intended to cover such changes, modifications, and equivalents.

[0173] The steps depicted and / or described are merely illustrative and may be omitted, combined, and / or performed in an order different from that depicted and / or described; the numbering of the depicted steps is merely for ease of reference and does not imply that any particular order is necessary or preferred.

[0174] The functions and / or steps described herein may be embodied in computer-usable data and / or computer-executable instructions executed by one or more computers and / or other devices to perform one or more functions described herein. In general, such data and / or instructions include routines, programs, objects, components, data structures, etc. that perform specific tasks and / or implement specific data types when executed by one or more processors in a computer and / or other data processing device. Computer-executable instructions may be stored in computer-readable media such as hard disks, optical disks, removable storage media, solid-state memory, read-only memory (ROM), random access memory (RAM), etc. As is understood, the functionality of such instructions may be combined and / or distributed as needed. In addition, the functionality may be embodied in firmware and / or hardware equivalents such as integrated circuits, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. Specific data structures may be used to more efficiently implement one or more aspects of the present disclosure, and it is conceivable that such data structures are within the scope of computer-executable instructions and / or computer-usable data described herein.

[0175] Although not required, one of ordinary skill in the art will appreciate that the various aspects described herein may be embodied as methods, systems, devices, and / or one or more computer-readable media storing computer-executable instructions. Thus, aspects may take the form of all-hardware embodiments, all-software embodiments, all-firmware embodiments, and / or embodiments combining software, hardware, and / or firmware aspects in any combination.

[0176] As described herein, various methods and actions may operate across one or more computing devices and / or networks. The functionality may be distributed in any manner, or may be located in a single computing device (eg, a server, client computer, user device, etc.).

[0177] Aspects of the present disclosure have been described with respect to illustrative embodiments thereof. Numerous other embodiments, modifications and / or variations within the scope and spirit of the appended claims may occur to one of ordinary skill in the art upon careful reading of this disclosure. For example, one of ordinary skill in the art will appreciate that the depicted and / or described steps may be performed in an order different from the order recited, and / or one or more of the illustrated steps may be optional and / or combined. Any and all features in the following claims may be combined and / or rearranged in any possible way.

[0178] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, modifications and / or equivalents of such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, changes and / or additions to the subject matter that will be readily apparent to those of ordinary skill in the art. For example, features illustrated and / or described as part of an embodiment may be used together with another embodiment to produce a further embodiment. Therefore, the present disclosure is intended to cover such changes, changes and / or equivalents.

Claims

1. A computer-implemented method comprising: presenting on a display of a computing device a graphical keyboard having a plurality of key regions, wherein the plurality of key regions include a first key region having a first set of keys and a second key region having a second set of keys, the first set of keys having a plurality of keys; receiving, by the computing device, a first input selecting a first selection area from among the plurality of key areas; determining, by the computing device, a first suggestion and a second suggestion based at least in part on the first input; as well as In response to the first input, an updated graphical keyboard having the plurality of key regions and a suggestion region is presented on the display of the computing device, wherein the suggestion region includes the first suggestion and the second suggestion.

2. The computer-implemented method of claim 1 , further comprising: receiving, by the computing device, a second input selecting a second selection area from among the plurality of key areas; determining, by the computing device, an updated suggestion based at least in part on the first input and the second input; as well as The updated suggestions are presented in the suggestion area of ​​the updated graphical keyboard.

3. The computer-implemented method of claim 2, wherein the updated suggestions are determined based on a ranking of a plurality of words, and wherein the updated suggestions are suggested words that exactly match the first input and the second input.

4. The computer-implemented method of claim 3, wherein the plurality of words are further ranked based on previous user interactions with the plurality of words.

5. The computer-implemented method of claim 3, wherein the updated suggestion is a suggested word that partially matches the first input and the second input, and wherein the suggested word begins with characters associated with the first input and the second input.

6. The computer-implemented method of claim 3, wherein the updated suggestion is a suggested phrase that partially matches the first input and the second input, and wherein the suggested phrase begins with characters associated with the first input and the second input.

7. The computer-implemented method of claim 1, wherein the updated graphical keyboard includes a sequence area, the method further comprising: determining a first determined symbol from a plurality of symbols based on the first selection area, and wherein the plurality of symbols includes a first symbol corresponding to the first key area and a second symbol corresponding to the second key area; as well as The determination symbol is presented in the sequence area of ​​the updated graphical keyboard.

8. A computer-implemented method as claimed in any preceding claim, wherein the first selection area is the first key area, the method further comprising: receiving, by the computing device, a second input selecting the second key region; determining, by the computing device, a second determined symbol from the plurality of symbols based on the second input, wherein the second determined symbol is different from the first determined symbol; as well as The second determination symbol is presented in the sequence area of ​​the updated graphical keyboard.

9. The computer-implemented method of claim 8, wherein the first deterministic symbol is a first shape having a first color and the second deterministic symbol is the first shape having a second color.

10. The computer-implemented method of any preceding claim, wherein the plurality of key regions includes a third key region having a third set of keys, and wherein the third set of keys is different from the first set of keys and the second set of keys.

11. The computer-implemented method of claim 10, wherein the plurality of key regions includes a fourth key region having a fourth set of keys.

12. A computer-implemented method as described in claim 11, wherein the first key set is keys located in the upper left quadrant of the graphical keyboard, wherein the second key set is keys located in the upper right quadrant of the graphical keyboard, wherein the third key set is keys located in the lower left quadrant of the graphical keyboard, and wherein the fourth key set is keys located in the lower right quadrant of the graphical keyboard.

13. The computer-implemented method of any one of claims 1 to 9, wherein the first set of keys are keys located in a left column of the graphical keyboard, and wherein the second set of keys are keys located in a right column of the graphical keyboard.

14. A computer-implemented method as claimed in any preceding claim, wherein the computer device comprises a first button and a second button, and the first input is received by a user pressing the first button or the second button.

15. The computer-implemented method of any preceding claim, further comprising: receiving a gesture via a sensor coupled to the computing device; as well as In response to the gesture, the first suggestion is selected.

16. A computer implemented method as claimed in any preceding claim, wherein the first key area has a greater number of keys than the second key area.

17. The computer-implemented method of any preceding claim, wherein the first suggestion is a word, and wherein the second suggestion is a phrase.

18. A computer device comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the computer device to perform operations comprising: presenting on a display of a computing device a graphical keyboard having a plurality of key regions, wherein the plurality of key regions include a first key region having a first set of keys and a second key region having a second set of keys; receiving a first input selecting a first selection area from among the plurality of key areas; determining a first suggestion and a second suggestion based at least in part on the first input; and In response to the first input, an updated graphical keyboard having the plurality of key regions and a suggestion region is presented on the display of the computing device, wherein the suggestion region includes the first suggestion and the second suggestion.

19. The computer apparatus of claim 18, wherein the updated graphical keyboard includes a sequence area, wherein the operations further comprise: receiving a second input selecting a second selection area from the plurality of key areas; determining an updated suggestion based at least in part on the first input and the second input; determining a first determined symbol associated with the first input from a plurality of symbols based on the first input, and wherein the plurality of symbols includes a first symbol corresponding to the first key area and a second symbol corresponding to the second key area; determining, based on the second input, from the plurality of symbols, a second determined symbol associated with the second input; presenting the first determination symbol and the second determination symbol in the sequence area of ​​the updated graphical keyboard; as well as The updated suggestions are presented in the suggestion area of ​​the updated graphical keyboard.

20. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: presenting on a display of a computing device a graphical keyboard having a plurality of key regions, wherein the plurality of key regions include a first key region having a first set of keys and a second key region having a second set of keys; receiving a first input selecting a first selection area from among the plurality of key areas; determining a first suggestion and a second suggestion based at least in part on the first input; as well as In response to the first input, an updated graphical keyboard having the plurality of key regions and a suggestion region is presented on the display of the computing device, wherein the suggestion region includes the first suggestion and the second suggestion.

21. A computing system comprising: one or more processors; as well as one or more memory devices storing instructions executable to cause the one or more processors to perform operations comprising: Rendering a graphics array of input features on a display component, the graphics array of input features comprising a first plurality of input features associated with the first region and a second plurality of input features associated with a second region; determining a region sequence code based on one or more inputs received from an input component, the region sequence code describing a sequence including one or more selections of the first region or the second region; One or more suggestion inputs are generated based on the region sequence encoding.

22. The system of claim 21, wherein the input feature corresponds to a language token, and wherein the one or more suggested inputs include suggested words of a language.

23. The system of claim 21 or 22, wherein the first area corresponds to a first zone of a graphical keyboard, and wherein the second area corresponds to a second zone of the graphical keyboard.

24. The system of any one of claims 21 to 23, wherein the one or more inputs include inputs associated with input signals assigned to the first zone and the second zone, respectively.

25. The system of claim 24, wherein the input signal corresponds to one or more physical toggle switches.

26. The system of claim 24, wherein the input signals correspond to areas on the touch screen that respectively cover the first area and the second area.

27. The system according to any one of claims 24 to 26, wherein the input signals correspond to peripheral components, respectively.

28. The system of any one of claims 21 to 27, wherein generating one or more suggestion inputs based on the region sequence encoding comprises: Inputting the region sequence code into a machine learning model; as well as The one or more suggested inputs are generated using the machine learning model.

29. The system of claim 28, wherein using the machine learning model to generate the one or more suggested inputs comprises: A probability distribution corresponding to the one or more suggestion inputs is generated.

30. The system of claim 28, wherein the machine learning model comprises a natural language model.

31. The system of any one of claims 28 to 30, wherein the machine learning model comprises one or more Transformer architectures.

32. A system as described in any of claims 28 to 31, wherein the machine learning model is trained on an unsupervised dataset for a target keyboard layout based on region sequences associated with input of symbols corresponding to words in a vocabulary.

Images