Machine learning based context aware correction for user input recognition

Abstract

Apparatuses and methods related to input recognition are provided. A method includes receiving an input by a user during interaction with a computing device. The method further includes receiving a transcription of the input from an input recognition model. The method further includes identifying a candidate term for replacement in the transcription. The candidate term may have been wrongly transcribed. The method additionally includes accessing a plurality of pairs of erroneously transcribed terms and non-common terms based on the candidate terms. These unusual terms may be wrongly transcribed, and these wrongly transcribed terms are incorrect versions of these unusual terms, which have been generated by a machine learning model. The method further includes replacing the candidate term with a non-common term in the transcription based on the erroneously transcribed term and the non-common term of the plurality of pairs.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS / INCORPORATED BY REFERENCE

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 363,319 filed on April 21, 2022, which is hereby incorporated by reference in its entirety. Background Art

[0003] Many modern computing devices, including mobile phones, personal computers, and tablet computers, include on-device input recognition capabilities. These devices can capture and recognize text or voice input and provide the recognized text or voice to one or more downstream applications.

[0004] The amount of computing resources available to mobile phone applications is limited. Therefore, enabling users to access effective input recognition tools while utilizing low computing resources is a significant technical improvement. Summary of the invention

[0005] When the input includes terms that are not recognized as common terms, the accuracy of on-device recognition of user input may be challenging. For example, when the input is in speech form, the way one or more phrases are pronounced may depend largely on the speaker. In some cases, phonetic variations in phrases, or spelling errors in terms, grammatical preferences, etc. may be personal habits of the user (e.g., names, locations, proper nouns entered by the user). In other cases, the use of certain terms may pose privacy challenges, especially when generating user-specific models. For example, medical terms such as "hernia" or "hemangioma" may be difficult to recognize in speech using an aggregated speech recognition model, and such medical terms may also be considered highly personalized content.

[0006] Some prior art techniques for improving input recognition involve user feedback. For example, a user may be prompted to provide terms as search terms, and personalization may be performed based on such user feedback. However, such personalization cannot be achieved in real time as speech is recognized. Other techniques may rely on user-generated rewrites of words or phrases to identify correctly transcribed terms. However, such user-generated rewrites may not be sufficient for infrequently occurring terms and / or terms specific to a particular domain. In addition, for example, a user may speak a term with different accents, dialects, etc., and it may be difficult to distinguish between different speech expressions of a particular term. Therefore, there is a need for an on-device input recognition tool that includes a context-aware correction module for performing automatic input recognition when performing input recognition at a device.

[0007] In one aspect, a computer-implemented method is provided. The method includes receiving, by an on-device system running on a computing device, input from a user during interaction with a computing device. The method further includes receiving a transcription of the input from an input recognition model. The method additionally includes identifying, by the on-device system, a candidate term for replacement in the transcription of the input, wherein the candidate term may have been incorrectly transcribed. The method also includes accessing, by the on-device system and based on the candidate term, a plurality of pairs of incorrectly transcribed terms and uncommon terms, wherein the uncommon terms may have been incorrectly transcribed, and wherein the incorrectly transcribed terms are incorrect versions of the uncommon terms, which have been generated by a machine learning model. The method further includes replacing the candidate term with an uncommon term in the transcription of the input, by the on-device system and based on the plurality of pairs of incorrectly transcribed terms and uncommon terms.

[0008] On the other hand, a computing device is provided. The computing device includes one or more processors and a data storage device. The data storage device stores computer executable instructions, which, when executed by one or more processors, cause the computing device to perform functions. The functions include receiving input from a user during interaction with the computing device by an on-device system running on the computing device. The functions further include receiving a transcription of the input from an input recognition model. The functions further include identifying, by the on-device system, a candidate term for replacement in the transcription of the input, wherein the candidate term may have been transcribed incorrectly. The functions also include accessing, by the on-device system and based on the candidate terms, multiple pairs of incorrectly transcribed terms and uncommon terms, wherein uncommon terms may be incorrectly transcribed, and wherein the incorrectly transcribed terms are incorrect versions of uncommon terms, which have been generated by a machine learning model. The functions further include replacing the candidate terms with uncommon terms in the transcription of the input by the on-device system and based on multiple pairs of incorrectly transcribed terms and uncommon terms.

[0009] On the other hand, a product is provided. The product includes one or more computer-readable media, on which are stored computer-readable instructions, which, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include receiving input from a user during interaction with the computing device by a system on a device running on the computing device. The functions further include receiving a transcription of the input from an input recognition model. The functions further include identifying a candidate term for replacement in the transcription of the input by the system on the device, wherein the candidate term may have been transcribed incorrectly. The functions also include accessing multiple pairs of incorrectly transcribed terms and uncommon terms by the system on the device and based on the candidate terms, wherein the uncommon terms may be transcribed incorrectly, and wherein the incorrectly transcribed terms are incorrect versions of uncommon terms, which have been generated by a machine learning model. The functions further include replacing the candidate terms with uncommon terms in the transcription of the input by the system on the device and based on multiple pairs of incorrectly transcribed terms and uncommon terms.

[0010] In another aspect, a system is provided. The system includes: means for receiving, by an on-device system running on a computing device, input from a user during interaction with a computing device; means for receiving a transcription of the input from an input recognition model; means for identifying, by the on-device system, a candidate term for replacement in the transcription of the input, wherein the candidate term may have been incorrectly transcribed; means for accessing, by the on-device system and based on the candidate term, a plurality of pairs of incorrectly transcribed terms and uncommon terms, wherein the uncommon terms may have been incorrectly transcribed, and wherein the incorrectly transcribed terms are incorrect versions of the uncommon terms that have been generated by a machine learning model; and means for replacing, by the on-device system and based on the plurality of pairs of incorrectly transcribed terms and uncommon terms, the candidate term in the transcription of the input with the uncommon term.

[0011] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An overall system of a context-aware correction model for input recognition according to an example embodiment is shown.

[0013] Figure 2 An example input recognition system is shown in accordance with an example embodiment.

[0014] Figure 3 Example context-aware correction for input recognition according to example embodiments is shown.

[0015] Figure 4 An example training phase of a neural network for a context-aware correction model for input recognition according to an example embodiment is shown.

[0016] Figure 5 is an example neural network for a context-aware correction model for input recognition according to an example embodiment.

[0017] Figure 6 is a diagram illustrating the training phase and the inference phase of a machine learning model according to an example embodiment.

[0018] Figure 7 A distributed computing architecture according to an example embodiment is depicted.

[0019] Figure 8 is a block diagram of a computing device according to an example embodiment.

[0020] Fig. 9 is a flowchart of a method according to an example embodiment.

[0021] Fig.10 is a flow chart of a correction method according to an example embodiment. DETAILED DESCRIPTION

[0022] Overview

[0023] The examples described herein relate to context-aware correction based on machine learning for input recognition (e.g., automatic speech recognition (ASR)). Specifically, a trained machine learning model (e.g., a convolutional neural network) can identify a term that may be incorrectly transcribed in speech transcribed by an ASR system, and replace the term that may be incorrectly transcribed with another term in the transcribed speech. For example, pairs of uncommon terms and associated incorrectly transcribed terms can be generated in advance to allow transcription to be improved substantially in real time. In some embodiments, uncommon terms can be terms observed during a user's previous interaction with the device so as to capture related terms without overburdening the system. In some embodiments, uncommon terms can be terms that may be incorrectly transcribed during transcription. For example, uncommon terms can be terms associated with a high probability of being incorrectly transcribed. In some embodiments, uncommon terms can be generated by a suitably trained machine learning algorithm. In addition, for example, uncommon terms can be corrected versions of incorrectly transcribed terms (e.g., previously corrected by a user and / or selected by a user). The incorrectly transcribed term may be incorrect or may be an incorrectly transcribed version of an uncommon term. In some embodiments, the incorrectly transcribed term can be generated using a machine learning model. For example, one or more machine learning algorithms may be used to identify and correct incorrect versions of uncommon terms. In some embodiments, the incorrectly transcribed terms are generated before the input recognition system running on the device generates them. Generally speaking, the incorrectly transcribed terms are possible errors that may occur during the transcription process.

[0024] The generated incorrectly transcribed terms remain on the device, thereby protecting the privacy of the user's data. In addition, for example, data generation and model training can be performed entirely on the user's device. This has significant computational and privacy benefits for large service providers, where there may be millions of users submitting billions of terms every hour.

[0025] The generated incorrectly transcribed terms are an extension of the capabilities of existing input recognition systems. The modified input recognition system uses terms that are familiar to the user, and thus the total number of incorrectly transcribed terms that the device must consider is limited to a small number. This reduces consumption of battery power and computing power, especially on resource-constrained mobile devices. The technique provides improved latency performance and significantly improved privacy properties.

[0026] The generated incorrectly transcribed terms may be applied to an upstream input recognition system. This ensures that input recognition can be performed on the server or on the device according to the best resources available. For example, available network resources may allow the device to communicate with the server. However, in situations where the network is unavailable and / or network bandwidth is limited, a modified on-device input recognition system may be used. Additionally, for example, input recognition may be performed at a remote server, while the correction module may be applied by the on-device system.

[0027] There are several advantages to applying the correction module at the device. For example, the generated incorrectly transcribed terms do not leave the device (e.g., not shared with a server, other devices, etc.) and are used for processing on the device. This has several advantages. For example, by limiting the content provided to the device, any user information that can be used for input recognition is protected by appropriate privacy and / or security controls. For example, the content can be encrypted, stored in a dedicated memory location on the device, and so on. These features also enhance the functionality of the input recognition system because the input recognition can be appropriately personalized to a specific user, user preferences, etc. based on the user's control over the degree of personalization. In addition, for example, the user can maintain control over what information is collected about the user, how to use the information, and what information is provided to the user. Processing on the device has advantages because it enables faster processing, lower power consumption, lower latency, and less network data transmission. In addition, for example, processing on the device can continue to perform automatic input recognition when the server is unavailable, the network is unavailable, the secure network is unavailable, and / or the network bandwidth is limited.

[0028] In addition to the above description, the user may be provided with controls that allow the user to make a choice about whether and when the system, program or feature described herein can collect user information (e.g., information about the user's social network, social contacts, or activities, user preferences or the user's current location, etc.) and whether to send content or communication from the server to the user. In addition, certain data may be processed in one or more ways before it is stored or used, so that personal data is removed, protected, encrypted, etc. For example, the identity of the user may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (such as to the city, zip code, or state level) when location information is obtained so that the user's specific location cannot be determined. Therefore, the user can control what information is collected about the user, how the information is used, and what information is provided to the user. In addition to user control, in embodiments where user information is used for input recognition, such user information is limited to the user's device and will not be shared with the server and / or other devices. In addition, for example, the user information may be deleted after use. For example, with the user's consent to use such data, the data may be used to determine incorrectly transcribed terms, which may then be securely stored on the device and not shared with other devices, servers, etc.

[0029] In some examples, the trained machine learning models may operate on a variety of computing devices including, but not limited to, mobile computing devices (e.g., smart phones, tablet computers, cellular phones, laptop computers), fixed computing devices (e.g., desktop computers), and server computing devices.

[0030] A machine learning model such as a convolutional neural network can be trained using training data (e.g., speech data) to perform one or more aspects as described herein. In some examples, the neural network can be arranged as an encoder / decoder neural network.

[0031] The trained machine learning model can process input data to predict output data including one or more words and / or phrases associated with the input data. In one example, (a copy of) the trained neural network can reside on a mobile computing device. In some embodiments, the mobile computing device can include a microphone capable of capturing input voice data. In response, the trained neural network can generate predicted output words associated with the input voice data.

[0032] In some embodiments, a first trained machine learning model may perform input recognition to generate a transcription of the input, and a second trained machine learning model may perform automatic correction of the transcription. Although the automatic correction of the transcription is performed by the on-device system, the input recognition may be performed by a remote server. In such embodiments, the on-device system may receive the input and send the input to the remote server. The remote server may apply the first trained machine learning model to generate a transcription of the input. Subsequently, the on-device system may receive the transcription from the remote server and perform automatic correction of the transcription.

[0033] Thus, the techniques described herein can improve input recognition by applying a context-aware correction module, thereby enhancing the actual and / or perceived quality of input recognition. Enhancing the actual and / or perceived quality of input recognition can provide benefits to downstream applications (e.g., voice-enabled applications) that rely on the output of the input recognition system. These techniques are flexible and can therefore apply a wide variety of user inputs, such as human voice, including various languages, dialects, and accents.

[0034] Local speech correction

[0035] An on-device correction module is described, wherein the correction module is context-aware. For example, the correction module can be trained using network-scale data synthetically generated via a noisy channel simulator. The model can then be fine-tuned for various applications. This technique can enable clients to fine-tune the same model, and this can enable transfer learning while limiting information leakage between clients.

[0036] Figure 1An overall system 100 of a context-aware correction model for input recognition according to an example embodiment is shown. Device 110 can receive input from a user during interaction with device 110 by an on-device system 105 running on device 110. Device 110 may include a computing device such as a laptop computer, a desktop computer, a smart TV, an electronic reading device, a streaming content device, a game console, a tablet device, or other related computing devices configured to execute software instructions and applications. The voice input spoken by the user can be a voice command to the application. For example, the application can be a media playback application for playing music, and the voice command can be an instruction to play a certain song. For another example, the voice command can be a search query to search the music library of the media playback application.

[0037] In another example, the application may be a text editor, and the voice command may be a dictation of input text. Additionally, for example, the voice command may be a command to perform one or more operations on the computing device. For example, the computing device may be a mobile device, and the voice command may be an instruction to open an application (e.g., a map, a contact, a browser, etc.), find directions to a location, initiate a voice or video call to a user in a contact list, send instructions to a digital assistant device (e.g., a home assistant device), etc. As another example, the computing device may be associated with a controller for an autonomous vehicle, and the voice command may be an instruction related to the operation of the vehicle.

[0038] In some embodiments, the on-device system 105 may be configured to run on an operating system of the device 110. The on-device system 105 may include an interface (e.g., via an application programming interface (API)) to communicate with one or more applications on the device 110. For example, the on-device system 105 may communicate with a first application via a first application API 115a, and may communicate with a second application via a second application API 115b.

[0039] The term "application" as used herein may be any computer program configured to interact with a user of device 110. Example applications may include a search application, an email application, a text messaging application, an instant messaging application, a web browsing application, a mapping application, a media playback application, a weather application, a phone application, a video communication application, a camera application, an application associated with a service provider (e.g., finance, insurance, etc.), an application associated with a digital assistant (e.g., a home assistant), or any other application configured to receive user input such as voice audio input, numeric text input, alphanumeric input, character input, and / or digital image input.

[0040] The term "interaction" may broadly refer to any active and / or passive activity performed by a user using device 110 or an application on device 110. For example, the interaction may involve viewing content, listening to content, inputting, editing and / or modifying content (e.g., via keyboard, mouse, tapping, etc.), sensory interaction (e.g., tactile, visual, auditory, haptic, etc.), scrolling interaction, voice interaction, user selection, etc. In some embodiments, the interaction may not be a direct interaction between the user and the content. For example, a user may listen to songs of a particular genre, or watch a movie of a particular genre. The computing device may determine the user interaction with a particular song of a particular genre as an interaction with songs of the same genre in the library. Similarly, the computing device may determine the user interaction with a particular movie of a particular genre as an interaction with movies of the same genre in the library. As another example, the user interaction with an email may be determined as an interaction with an entire email chain, and / or a multiple email exchange with a particular sender of an email.

[0041] In some embodiments, the user interaction may be an interaction with a digital assistant (e.g., an intelligent digital assistant). For example, a user may send a voice command such as, for example, "turn on the lights in the den", "play music by Jenna K.", "play the second episode of the series I watched yesterday", "set the thermometer settings to home", and the like. In some embodiments, the user interaction may be an interaction with a search assistant. For example, a user may enter text into a search field of a web browser. As another example, a user may use a voice command to enter a search term such as, for example, "search for the nearest Italian restaurant that was reviewed in the local daily last weekend".

[0042] In some embodiments, the user interaction can be an interaction with a map application. For example, the user can enter a street address as text input in an address input field of a mapping application. Additionally, for example, the user can use voice commands to enter a destination for a navigation application. For example, the user can say "take me home" or "find me a route with no tolls", "is there public transport to the Globe Theater", etc.

[0043] The on-device system 105 may interact with one or more hardware components 130 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a memory, an input and / or output device of the device 110). For example, the on-device system 105 may interact with one or more input method editors such as a voice editor to edit voice input from a microphone 135. In some embodiments, the voice input by the user may be captured by the microphone 135. The microphone 135 may be a part of the device 110, or may be an audio input device (e.g., a wired or wireless microphone) that is separate from the device 110 and communicatively linked to the device 110. For example, the user may activate the microphone 135 to enable voice dictation, or send a voice command to perform an action. In addition, for example, the on-device system 105 may interact with an input method editor for text input. Text input may be performed through a numeric keypad or a physical keyboard. For example, the touch screen of the device 110 may display a numeric keypad. In some embodiments, different numeric keypads corresponding to different languages ​​may be displayed. In addition, for example, different numeric keypads corresponding to different layouts, designs, etc. may be displayed.

[0044] The on-device system 105 may interact with an input recognition system 125. In some embodiments, the on-device system 105 may receive a transcription of a user input from the input recognition system 125. The input recognition system 125 may be configured to recognize user input. For example, the input recognition system 125 may be a speech recognition system configured to recognize speech. In addition, for example, the input recognition system 125 may be a text recognition system configured to recognize text. For example, the input recognition system 125 may include text recognition logic, programming instructions, or algorithms to manage the identification, extraction, and analysis of characteristics of text input. For example, the input recognition system 125 may execute comparator logic to compare the spatial characteristics of the text input with various model parameters in the spatial model and / or language model. For example, the spatial model may be used for text prediction by associating the spatial coordinates of letters or the spatial relationship between letters entered by typing, swiping, or gesturing.

[0045] In some embodiments, the input recognition system 125 may reside on the device 110. In some embodiments, the input recognition system 125 may be an interface (e.g., an application programming interface) at the device 110 for an input recognition system residing at a remote server. For example, a first trained machine learning model may perform input recognition to generate a transcription of the input, and a second trained machine learning model may perform automatic correction of the transcription. Although the automatic correction of the transcription is performed by the on-device system (e.g., the context-aware correction module 120), the input recognition may be performed by a remote server (e.g., the input recognition system 125) that interacts with the device 110 through an API. In such embodiments, the on-device system may receive user input and send the user input to the remote server. The remote server may apply the first trained machine learning model to generate a transcription of the input. Subsequently, the on-device system may receive the transcription from the remote server and perform automatic correction of the transcription.

[0046] Some embodiments relate to receiving transcription of input from an input recognition model. For example, an input recognition system 125, such as, for example, a speech recognition system, may transcribe speech input received by a microphone 135 and provide the transcription. The term "speech recognition" as used herein may generally refer to any process of recognizing audio input and converting the audio input into a text format. For example, a microphone 135 may receive speech input in the form of human speech, and the input recognition system 125 may transcribe the human speech into text. In some embodiments, the input recognition system 125 may include one or more machine learning models trained to recognize speech.

[0047] Identifying input (e.g., speech, text, etc.) may be a challenging task. For example, there may be challenges due to the inherent complexity of various languages ​​and / or dialects therein. In some embodiments, different individuals may vary in the way they speak and / or input text. For example, when a particular term is not recognized as a common term in a dictionary associated with a language, input recognition may be particularly challenging. Although input recognition systems can be trained to recognize different languages, accents, dialects, etc., such input recognition systems may still be unable to recognize words or phrases spoken and / or input by the user. In addition, for example, such input recognition systems may be located at a server far away from the device, and various data processing restrictions may limit the appropriate personalization of such systems, and / or network restrictions may limit access to such remote servers. Therefore, both for privacy and / or security control, and in order to achieve faster processing, lower power consumption, lower latency, less network data transmission, etc., on-device processing may be preferred.

[0048] In some embodiments, the on-device system 105 may identify candidate terms for replacement in the input transcription. For example, the on-device system 105 may perform one or more operations including scanning the transcription, filtering out common terms, performing contextual analysis, etc. The on-device system 105 may then determine that the transcription is accurate (within an appropriate threshold of permissible accuracy) and make no corrections. In some embodiments, the on-device system 105 may identify one or more candidate terms that may have been transcribed incorrectly. For example, the on-device system 105 may identify one or more uncommon terms that may have been transcribed incorrectly, one or more terms that have been previously transcribed incorrectly, one or more terms that have been previously corrected by the user in a past interaction with the computing device, and the like.

[0049] In some embodiments, the accuracy of transcription of user input (e.g., speech-to-text conversion, text input via a keyboard, etc.) can be enhanced based on the context of the input (e.g., an email application, a mapping application, a home assistant, a search query, etc.). For example, phonetically similar words can be transcribed into two different transcribed versions based on the context of the speech. Therefore, the on-device system 105 can include a context-aware correction model 120 that can correct the transcription of the input by the input recognition system 125. For example, the context-aware correction model 120 can interact with one or more applications on the device 110 (e.g., interact with a first application through a first application API 115a, and interact with a second application through a second application API 115b), understand the underlying context of the user input, and correct the transcription terms generated by the input recognition system 125.

[0050] In some embodiments, the system 105 on the device accesses multiple pairs of mistranscribed terms and uncommon terms based on candidate terms, wherein uncommon terms may be mistranscribed, and wherein the mistranscribed terms of the uncommon terms in the multiple pairs are generated by a machine learning model. In some embodiments, the system on the device may have observed uncommon terms in the multiple pairs in one or more past interactions between the user and the computing device. In some embodiments, the uncommon terms in the multiple pairs may have been generated synthetically (e.g., based on aggregate statistics of user input, trained machine learning models, etc.). In addition, for example, the uncommon terms may be corrected versions of the mistranscribed terms (e.g., previously corrected by the user during the automatic correction process and / or selected by the user). The mistranscribed terms may generally be mistranscribed versions of uncommon terms. In some embodiments, the mistranscribed terms may be generated using a machine learning model. For example, one or more machine learning algorithms may be used to identify and correct incorrect alternatives. In some embodiments, the mistranscribed terms are generated before the input recognition system running on the device is generated. Generally speaking, the mistranscribed terms are possible errors that may occur during the transcription process. As described herein, pairs of uncommon terms and associated mistranscribed terms may be generated in advance to allow for immediate improvement of transcription.

[0051] In some embodiments, based on a plurality of pairs of mis-transcribed terms and uncommon terms, the on-device system 105 may replace a candidate term with an uncommon term in the input transcription. For example, the on-device system 105 may compare the candidate term with one or more mis-transcribed terms, and then select (e.g., based on the ranking of the pairs) an uncommon term from the pairs of mis-transcribed terms and uncommon terms. The on-device system 105 may then replace the candidate term with the selected uncommon term.

[0052] Figure 2 An example input recognition system 200 is shown according to an example embodiment. In some embodiments, speech input 210 spoken by a user 205 during interaction with a computing device 220 may be received by an on-device system 215 running on the computing device 220. The speech input 210 may be received in the form of an audio signal 225. Feature extraction 230 may be configured to extract one or more features of the audio signal 225. The acoustic model 235 may be configured to associate the audio signal 225 with the relationship between phonemes or other linguistic properties that form the speech audio. For example, the acoustic model 235 may be configured to identify and associate certain received utterances that exhibit acoustic characteristics that are consistent with the acoustics associated with a spoken word or phrase.

[0053] The language model 240 may be configured to specify or identify certain word combinations or sequences. In some implementations, the language model 240 may be configured to generate a word sequence probability factor that may be used to indicate the likely occurrence or presence of a particular word sequence or word combination. The identified word sequences may correspond primarily to sequences that are specific to a speech corpus rather than a written corpus.

[0054] The speech recognition model 245 may be configured to receive inputs from the acoustic model 235 and the language model 240 to generate a transcription of the speech input 210. For example, the speech recognition model 245 may be configured to include speech recognition logic, programming instructions, and / or algorithms that are executed by one or more processors to transcribe the speech input 210. For example, the speech recognition model 245 may execute program code to manage the identification, extraction, and analysis of characteristics of the received audio signal 225. In addition, the speech recognition model 245 may execute comparator logic to compare the characteristics of the received audio signal 225 with various model parameters stored in the acoustic model 235 and the language model 240. The result of the comparison may produce a text transcription output that substantially corresponds to the speech input 210 provided by the user 205 of the computing device 220.

[0055] In some embodiments, the on-device system 215 may access a plurality of pairs of mis-transcribed terms and uncommon terms. The on-device system 215 may have observed uncommon terms in the plurality of pairs in one or more past interactions of the user 205 with the computing device 220. Generally speaking, an "uncommon term" as used herein may refer to any term that may be mis-transcribed during the speech-to-text transcription process. In some embodiments, the uncommon term may have different phonetic versions and / or may be different transcribed versions of the phonetic term.

[0056] In some embodiments, user 205 may have viewed one or more documents related to medicine, and on-device system 215 may have extracted uncommon terms that appear in the one or more documents related to medicine. For another example, user 205 may have listened to one or more songs related to a genre of music, and on-device system 215 may have extracted uncommon terms that appear in the one or more songs, or transcriptions thereof. Additionally, for example, user 205 may have interacted with a text editor to enter and / or modify text, and on-device system 215 may have extracted uncommon terms that appear in the text.

[0057] The incorrectly transcribed terms of the uncommon terms in the plurality of pairs may have been generated by a machine learning model. In general, "incorrectly transcribed terms" may refer to different versions of the uncommon terms, where the different versions correspond to possible phonetic variations of the uncommon terms when spoken in speech; different text versions; incorrectly spelled versions; and / or possible incorrectly transcribed versions of the uncommon terms. The incorrectly transcribed terms may be synthetically generated by a trained machine learning model. In some embodiments, the incorrectly transcribed terms may be generated based on training data using speech spoken by different individuals.

[0058] In some embodiments, the on-device system 215 may identify candidate terms for replacement in the input transcription. For example, the on-device system 215 may identify a candidate term 250 that may have been erroneously transcribed by the speech recognition model 245. Thus, the context-aware correction model 255 may be configured to determine whether the candidate term 250 is erroneously transcribed, and if so, replace the candidate term 250 with the corrected term 260 in the input transcription. In some embodiments, the context-aware correction model 255 may be configured to communicate with the computing device 220 to identify one or more uncommon terms observed by the on-device system 215 in one or more past interactions of the user 205 with the computing device 220.

[0059] although Figure 2 The various components of are shown as being on the computing device 220 for illustrative purposes only, but various additional and / or alternative embodiments are also contemplated. For example, although the context-aware correction model 255 resides on the computing device 220, one or more of the other components may also reside on the computing device 220, at a remote server, or both. For example, the speech recognition model 245 may reside on the computing device 220. In some embodiments, the speech recognition model 245 may reside at a remote server (e.g., a cloud server). In addition, for example, some portions of the speech recognition model 245 may reside on the computing device 220, while other portions may reside at a remote server.

[0060] For example, the on-device system 215 may receive the audio signal 225 and send the audio signal 225 to a remote server for processing. Feature extraction 230 may be performed by the remote server. Additionally, for example, at the remote server, the speech recognition model 245 may utilize the acoustic model 235 and the language model 240 to generate a transcription based on the audio signal 225 and send the transcription to the computing device 220. The on-device system 215 may then identify the candidate term 250, apply the context-aware correction model 255, and identify the corrected term 260 to correct the transcription.

[0061] although Figure 2An example embodiment for speech recognition is described, but similar methods may also be applied to text recognition. For example, the speech recognition model 245 may be a text recognition model that can utilize a keyboard model to recognize text input. In some embodiments, the keyboard model may include a spatial model, a language model, a keyboard input mode editor, and the like. For example, the keyboard model may be configured to receive touch and / or physical input corresponding to letters, numbers, symbols, emojis, and / or characters. Incorrectly transcribed terms may include variations in spelling, typical spelling errors, common grammatical errors, variations in capitalization, and the like.

[0062] Figure 3 An example context-aware correction 300 for input recognition according to an example embodiment is shown. For example, a user 305 may utter speech 340 while interacting with a device 310, such as, "find me items related to nurnia" including the term "Nurnia". Although the device 310 is shown as a mobile device, the device 310 may also be any device configured to interact with the user 305 (e.g., a smart TV, a digital content delivery device). In some embodiments, the device 310 may receive the speech spoken by the user 305, and the speech recognition system may transcribe the spoken speech into a transcription of the speech input, such as, "find me items related to nurnia". In some embodiments, the context-aware correction model 325 may identify "nurnia" as a candidate term for replacement.

[0063] In some embodiments, the context-aware correction model 325 may access multiple pairs of incorrectly transcribed terms and uncommon terms. For example, the context-aware correction model 325 may access an on-device repository 360 of pairs of incorrectly transcribed terms and uncommon terms. As an illustrative example, pairs of incorrectly transcribed terms and uncommon terms may be "(haarnia, hernia)", "(narnia, hernia)" and "(hair near, hernia)", wherein the incorrectly transcribed terms "haarnia", "narnia" and "hair near" are paired with the uncommon term "hernia". As shown in box 355, the context-aware correction model 325 may compare the transcribed term "nurnia" 350 with the incorrectly transcribed term, and determine that "nurnia" 350 matches the incorrectly transcribed term "narnia". The term "match" as used herein may generally refer to a match within a similarity threshold. In some embodiments, the term "match" may represent a complete match.

[0064] In some embodiments, the context-aware correction model 325 can replace candidate terms with uncommon terms in the input transcription based on multiple pairs of incorrectly transcribed terms and uncommon terms. For example, when determining that "nurnia" 350 matches the incorrectly transcribed term "narnia", the context-aware correction model 325 can determine that the uncommon term "hernia" paired with "narnia" as "(narnia, hernia)" is the correct transcription of the term "nurnia" 350. For example, the selection of a pair can be based on the confidence level of the pair. Therefore, the uncommon term "hernia" 365 is used to replace the candidate term "nurnia" 350 in the transcribed text. For example, the context-aware correction model 325 corrects the transcription "find me items related to nurnia (find items related to nurnia for me)" generated by the speech recognition system to "find me itemsrelated to hernia (find items related to hernia for me)". Although this example shows the replacement of a single term "nurnia", the same technology can also be applied to correct more than one candidate term in the transcription.

[0065] In some embodiments, replacement of a candidate term involves comparing, by the on-device system, the candidate term with one or more incorrectly transcribed terms from a plurality of pairs. Generally, a first comparison may be made between the candidate term and a plurality of incorrectly transcribed terms based on similarity determination or matching, and a second comparison may be made between pairs including incorrectly transcribed terms based on respective confidence levels of the pairs. In some embodiments, a final selection of a pair may be based on a combination of the first comparison and the second comparison.

[0066] In some embodiments, the context-aware correction model 325 may be configured to parse (e.g., observe) the text that the user 305 is viewing and / or editing. This may include text input of mails, short messages, query terms and / or text that the user 305 is viewing on the device 310, such as, for example, web documents, inbound messages, text within applications, etc. The content does not leave the device (e.g., not shared with a server, other devices, etc.), and is used for processing on the device. This has several advantages. For example, by limiting the content provided to the device, any user information that can be used for input recognition is protected by appropriate privacy controls. These features also enhance the functionality of the input recognition system because the input recognition can be appropriately personalized to a specific user, user preferences, etc., based on the user's control over the degree of personalization. In addition, for example, the user can maintain control over what information is collected about the user, how the information is used, and what information is provided to the user. Processing on the device has advantages because it enables faster processing, lower power consumption, lower latency, and less network data transmission. In addition, for example, in the case where the server is unavailable, the network is unavailable, the secure network is unavailable, and / or the network bandwidth is limited, processing on the device can continue to perform automatic input recognition.

[0067] In some embodiments, identifying candidate terms for replacement in an input transcription by the on-device system may involve filtering common terms and retaining uncommon terms. For example, the context-aware correction model 325 may be configured to filter common terms and retain uncommon terms. The candidate terms may have been incorrectly transcribed.

[0068] For example, user 305 may have had past interactions with device 310. For example, user 305 may have viewed content related to medical information. Therefore, context-aware correction model 325 may have screened the uncommon term "hernia" that appeared in the content previously viewed by user 305. In some embodiments, context-aware correction model 325 may have generated one or more incorrectly transcribed terms (e.g., by using a trained machine learning model). The incorrectly transcribed term may be determined as a text version of a phonetically similar utterance of the uncommon term "hernia". For example, context-aware correction model 325 may have generated "haarnia", "narnia" and "hair near" as incorrectly transcribed terms corresponding to the uncommon term "hernia". Therefore, pairs of incorrectly transcribed terms and uncommon terms "(haarnia, hernia)", "(narnia, hernia)" and "(hair near, hernia)" may be stored in a repository 360 on the device.

[0069] In some embodiments, the past interaction of user 305 with device 310 may have included viewing media content. For example, user 305 may have viewed a movie named "Narnia", or may have listened to the soundtrack from the movie "Narnia". For another example, user 305 may have read the review of the movie "Nania". Therefore, the context-aware correction model 325 may have screened the uncommon term "narnia" that appeared in the content previously viewed by user 305. In some embodiments, the context-aware correction model 325 may have generated one or more incorrectly transcribed terms (e.g., by using a trained machine learning model). The incorrectly transcribed terms may be determined as a text version of a phonetically similar utterance of the uncommon term "narnia". For example, the context-aware correction model 325 may have generated "haarnia", "hernia" and "hair near" as incorrectly transcribed terms corresponding to the uncommon term "narnia". Thus, pairs of incorrectly transcribed terms and uncommon terms “(haarnia, narnia)”, “(hernia, narnia)”, and “(hair near, narnia)” may be stored in the on-device repository 360 .

[0070] In some embodiments, each of the pairs of mis-transcribed terms and uncommon terms stored in the on-device repository 360 may be associated with a respective confidence level indicating the similarity of the particular mis-transcribed term and the particular uncommon term paired with the particular mis-transcribed term. In some embodiments, the similarity may be a measure of phonetic similarity between the mis-transcribed terms and the uncommon terms.

[0071] The confidence level may be determined based on one or more factors. For example, the confidence level may be based on the source of the incorrectly transcribed term. In some embodiments, a user's voice sample may be used to select or deselect a term as an incorrectly transcribed term. Additionally, for example, one or more previous voice interactions of the user may be used to select or deselect a term as an incorrectly transcribed term. In such embodiments, the confidence level associated with a pair including a selected incorrectly transcribed term and an associated uncommon term may be determined to be high.

[0072] In some embodiments, the frequency of utterance of a user's incorrectly transcribed terms may be used to determine a confidence level. For example, when an incorrectly transcribed term is spoken with high frequency in the same manner, the confidence level associated with a pair including the incorrectly transcribed term and an associated uncommon term may be determined to be high. For example, a user may pronounce the uncommon term "Narnia" as "naaarnia," and the computing device may determine that the frequency of uttering "Narnia" as "naaarnia" is above a frequency threshold. Therefore, the confidence level associated with the pair (Naaarnia, Narnia) may be determined to be high. The frequency of utterance of a particular incorrectly transcribed term may be based on several factors, including term frequency, term frequency-inverse document frequency (TF-IDF), and the like.

[0073] In some embodiments, the frequency of user observation of uncommon terms may determine a confidence level. For example, a computing device may determine the frequency of occurrence of a particular uncommon term in content viewed by a user. The viewed content may be based on a single document, a single application, or multiple applications. Thus, based on a determination that a user observes an uncommon term with a high frequency, a confidence level associated with a pair comprising an uncommon term and an associated incorrectly transcribed term may be determined to be high. Similarly, based on a determination that a user observes an uncommon term with a low frequency, a confidence level associated with a pair comprising an uncommon term and an associated incorrectly transcribed term may be determined to be low. In general, the frequency of a particular uncommon term may be based on several factors, including term frequency, term frequency-inverse document frequency (TF-IDF), and the like.

[0074] In some embodiments, the confidence level for the pair may be based on the respective frequencies of occurrence of the mistranscribed term and the uncommon term. For example, the confidence level may be based on the joint distribution of the individual frequencies. In some embodiments, the confidence level associated with the pair may vary based on the underlying application and / or the context of the uncommon term. Additionally, for example, the confidence level associated with the pair may vary from one user to another. In some embodiments, the confidence level associated with the pair may be based on language, accent, geographic location, etc.

[0075] For example, the pair "(haarnia, hernia)", "(narnia, hernia)", and "(hair near, hernia)" may each be associated with a high confidence level, indicating that each of the incorrectly transcribed terms "haarnia", "narnia", and "hair near" has a high degree of similarity to the uncommon term "hair near". In some embodiments, the similarity may be a phonetic similarity. In another example, the pair "(hyena, hernia)" may be associated with a medium confidence level, indicating that the incorrectly transcribed term "hyena" has a medium degree of similarity to the uncommon term "hernia". Additionally, for example, the pair "(herein, hernia)" may be associated with a low confidence level, indicating that the incorrectly transcribed term "herein" has a low degree of similarity to the uncommon term "hernia".

[0076] Similarly, the pairs "(haarnia, narnia)", "(hernia, narnia)", and "(hair near, narnia)" may each be associated with a high confidence level, indicating that each of the incorrectly transcribed terms "haarnia", "hernia", and "hair near" has a high degree of similarity to the uncommon term "narnia". As another example, the pair "(naina, narnia)" may be associated with a medium confidence level, indicating that the incorrectly transcribed term "naina" has a medium degree of similarity to the uncommon term "narnia". Additionally, for example, the pair "(aria, narnia)" may be associated with a low confidence level, indicating that the incorrectly transcribed term "aria" has a low degree of similarity to the uncommon term "narnia".

[0077] In operation, user 305 may interact with device 310. In some embodiments, user 305 may interact with content 345. For example, user 305 may be a medical practitioner, and content 345 may be a real-time transcription of user 305's voice interaction with a patient. Thus, context-aware correction model 325 may compare the transcribed term "nurnia" 350 with the incorrectly transcribed term, and determine that "nurnia" 350 matches the incorrectly transcribed term "narnia". However, based on content 345 indicating a medical context, context-aware correction model 325 may determine that "hernia" 365 is the correct transcription of the transcribed term "nurnia" 350.

[0078] Some embodiments involve receiving, by a computing device, a second voice input spoken by a user during a second interaction with the computing device, wherein the second transcription of the input includes a candidate term. For example, during the second interaction, the user 305 may interact with a media playback application, and the content 345 may be a browser that displays content provided by the media playback application. For example, the user 305 may speak a voice 340 to submit a voice search query to the media playback application. Such embodiments involve comparing the candidate term with one or more second incorrectly transcribed terms, wherein the one or more second incorrectly transcribed terms are observed by the system on the device in one or more past interactions of the user with the second application of the computing device, and wherein the one or more second incorrectly transcribed terms are different from the one or more incorrectly transcribed terms. For example, the context-aware correction model 325 may compare the candidate term "nurnia" 350 with the incorrectly transcribed term, and determine that "nurnia" 350 matches the incorrectly transcribed term "narnia". However, based on the content 345 indicating the media playback context, the context-aware correction model 325 may determine that "narnia" is the correct transcription of the transcribed term "nurnia" 350. Such embodiments also involve replacing, by the on-device system, a candidate term in the input transcription with a second uncommon term based on the comparison, wherein the second uncommon term is paired with a second incorrectly transcribed term in the one or more second incorrectly transcribed terms. For example, replacing "nurnia" 350 with the second uncommon term "Narnia".

[0079] In general, determining the incorrectly transcribed terms, the pairs including the incorrectly transcribed terms, the confidence level associated with each pair, the acceptance threshold for replacing the pairs, one or more of which may vary from one user to another, from one application to another, etc. In addition, for example, such determinations may be made by a trained machine learning model. In addition, for example, such determinations may vary based on the application, the user, the user's geographic location, and / or the language used.

[0080] In some embodiments, user feedback may be incorporated into such determinations. In some embodiments, one or more past interactions of the user with the computing device may involve voice interactions. The uncommon terms in the plurality of pairs may be based on user confirmation of the terms based on the transcription of the voice interactions. For example, the ASR may transcribe speech spoken by the user and prompt the user to confirm that the transcribed terms are correct.

[0081] In some embodiments, one or more past interactions of the user with the computing device may involve interactions with a text editor. For example, the user may enter text in an electronic message, a short message, etc., and the computing device may prompt the user to confirm the corrected term and / or prompt the user to select the correct term from one or more candidate terms provided to the user. Thus, the uncommon term in the plurality of pairs may be based on the user's confirmation and / or selection of the text term in the text editor.

[0082] In some embodiments, the computing device may include a viewer interface. One or more past interactions of the user with the computing device may involve textual content provided by the viewer interface. For example, the user may browse documents on the web, browse libraries associated with media content, and the like. The uncommon terms presented in the plurality of pairs may present textual content, such as documents being browsed, music libraries, and the like.

[0083] Input recognition models are usually trained based on user preferences. However, the quality of such training depends on the quality of available logs of past user preferences. Generally speaking, maintaining such logs may require a large amount of memory space and also requires continuous training of the input recognition model, which may consume a lot of resources. However, a more efficient approach is to identify content that the user has viewed and filter out uncommon terms from the content, and store these terms in a local repository as potential correct terms in the context of input recognition.

[0084] Variations in a user's accent may be a source of errors in automatic speech recognition. A user's geographic or local accent may sometimes be determined by the user's location information. Additionally, for example, a user's ethnicity may indicate ethnic dialects (e.g., national dialects), variations in spelling, etc. For example, ethnic dialects may particularly indicate influences from the user's first language. Some variation factors may also be inferred from user information. In some embodiments, preferences provided by the user may be used. For example, a user may be prompted to speak a sentence, and the automatic speech recognition system may infer nuances and speech characteristics that are unique to the user.

[0085] In addition to the above description, the user may be provided with a control that allows the user to make a choice about whether and when the system, program or feature described herein can collect user information (e.g., information about the user's race, gender, social network, social contacts, or activities, user preferences or the user's current location, etc.) and whether to send content or communication from the server to the user. In addition, certain data may be processed in one or more ways before it is stored or used, so that personal data is removed, protected, encrypted, etc. For example, the identity of the user may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (such as to the city, zip code or state level) when location information is obtained so that the user's specific location cannot be determined. Therefore, the user can control what information is collected about the user, how the information is used, and what information is provided to the user. In addition to user control, in an embodiment where user information is used for input recognition, such user information is limited to the user's device and will not be shared with the server and / or other devices. In addition, for example, the user information may be deleted after use. For example, with the user's consent to use such data, the data may be used to determine pairs of terms including incorrect transcriptions and uncommon terms, and the determined pairs may be stored on the device. However, the source of the content related to the incorrectly transcribed terms and uncommon terms may not be stored. As another example, the user history records related to the past interactions of the device may not be stored.

[0086] Figure 4 An example training phase of a neural network 400 for a context-aware correction model for input recognition according to an example embodiment is shown. In some embodiments, initial training of the neural network 400 can be performed without user logs. In addition, domain adaptation for a particular application can be accomplished, for example, by synthesizing simulated errors (e.g., speech errors, context errors, etc.). For example, a YouTube video can be trained for creators, channels, artists, video names, etc. TM Domain adaptation of the context-aware correction model 450 can be performed, and the base model can be fine-tuned subsequently. For example, an initial version of the context-aware correction model 450 can be pre-trained remotely, and subsequent training can be performed at the computing device. As described herein, the input recognition system described herein can be trained and reside at a remote server, while the context-aware correction model 450 can reside at a local computing device.

[0087] As another example, a music application may be installed on a device, and the device may have access to a catalog of music. Thus, pronunciation variations of uncommon terms (e.g., artist names, song titles, portions of lyrics, etc.) may be generated as incorrectly transcribed terms. When a user interacts with the voice interface of the music application, the context-aware correction model may correct the output of the automatic speech recognition system to achieve accurate transcription of the spoken speech.

[0088] Additionally, for example, an email system may be installed on the device, and the device may access contact information associated with the email system. The device may also be capable of observing a user reading one or more messages, and uncommon terms may be identified from the content of such messages. Pronunciation variations of such uncommon terms (e.g., names, titles, key words or phrases, etc.) may be generated as incorrectly transcribed terms. Subsequently, when the user interacts with the voice interface of the email system (e.g., dictating a new message), the context-aware correction model may correct the output of the input recognition system to achieve accurate transcription of spoken speech.

[0089] As another example, an application may provide reviews for restaurants in a geographic area. Such an application may access restaurant names, addresses, menu items, chef names, dish names, and the like. The device may observe users browsing menus at a particular restaurant, and may identify uncommon terms from such browsing activity. Pronunciation changes of such uncommon terms may be generated as incorrectly transcribed terms. Subsequently, the user may interact with the voice interface of a restaurant review application or another application (e.g., a mapping application, an appointment booking application, a short messaging application, an email application, a calendar application, etc.), and the context-aware correction model may correct the output of the input recognition system to achieve accurate transcription of the spoken speech. For example, a user may use a mapping application to generate directions to a particular restaurant, and the context-aware correction model may achieve accurate recognition of restaurant names, street names, and the like. As another example, a user may use a short messaging application to send a text message to a contact to arrange a meeting at a particular restaurant that provides a particular dish, and the context-aware correction model may enable accurate recognition of restaurant names, dish types, and the like.

[0090] In some embodiments, the training of the machine learning model further includes receiving a corpus of documents. In some embodiments, the training of the machine learning model may be performed at a server remote from the computing device in which the context-aware correction model 450 resides. For example, the machine learning model may be pre-trained at a remote server. The document corpus 405 may be received by the pronunciation-based text API 410. The document corpus 405 may include multiple documents from the Internet. In addition, for example, the document corpus 405 may be an application-specific document, such as, for example, a content list for a media playback application; and / or literature specific to other application domains. In some embodiments, the documents within the document corpus 405 may be associated with one or more applications and / or concepts to which the document relates. For example, the document corpus 405 may include documents related to music, movies, medicine, law, philosophy, history, literature, etc.

[0091] In some embodiments, the training of the machine learning model further includes synthetically simulating one or more errors based on a corpus of web documents. In some embodiments, the synthetic simulation of one or more errors is based on a text-to-speech model using a noise channel simulator. For example, a text-to-speech (TTS) model 415 with noise and accent can be used based on the pronunciation-based text API 410. For example, the TTS model 415 can include training data, wherein the text is converted into speech under different controlled noise conditions to generate voice changes caused by noise conditions. For example, different forms of noise (e.g., random noise) can be artificially injected into the speech to destroy the speech and generate training data. In addition, for example, different forms of background noise (e.g., traffic noise, airport noise, wind, water, rain, background music, background conversations, etc.) can be artificially injected into the speech to destroy the speech and generate training data. The pronunciation-based text API 410 can apply the TTS model 415 to the document corpus 405 to generate a modified corpus 430 including text errors in the document corpus 405, wherein these errors are caused by noise conditions.

[0092] Additionally, for example, TTS model 415 may include training data in which text is converted to speech with different accents, dialects, etc. For example, a user may be prompted to read different text documents, and user variations and speech characteristics may be captured. Pronunciation-based text API 410 may apply TTS model 415 to document corpus 405 to generate a modified corpus 430 that includes text errors in document corpus 405, where the errors are caused by different accents, dialects, etc.

[0093] In some embodiments, the comprehensive simulation of one or more errors is based on a grapheme-to-phoneme conversion model that is configured to generate a pronunciation of a word based on a text version of the word. For example, the pronunciation-based text API 410 may receive the output of an automatic speech recognition (ASR) system and a grapheme-to-phoneme (G2P) conversion 420. For example, the G2P conversion 420 may include training data in which a grapheme sequence including letters is converted to a phoneme sequence representing a pronunciation of the grapheme sequence. The pronunciation-based text API 410 may apply the G2P conversion 420 to the document corpus 405 to generate a modified corpus 430 that includes text errors in the document corpus 405, where the errors are caused by different phoneme sequences representing the pronunciation of the grapheme sequence.

[0094] In some embodiments, the comprehensive simulation of one or more speech errors is based on a statistical phoneme model. For example, a text API 410 based on pronunciation can receive input from a statistical phoneme model 425. For example, a statistical phoneme model 425 can utilize various phoneme characteristics such as provided by an ASCII transcription of an International Phonetic Alphabet (IPA). The statistical phoneme model 425 can include training data based on different pronunciation methods. For example, the source of the airflow can be considered, such as whether the airflow originates from the lungs, tongue, glottis, etc. In some embodiments, the target of the airflow can be considered, such as, for example, whether the airflow is directed to the mouth or nose. The direction of the airflow can be considered, for example, whether the airflow is toward the target or away from the target. Additional and / or alternative phoneme characteristics can be modeled by a statistical phoneme model 425. Therefore, a text API 410 based on pronunciation can apply a statistical phoneme model 425 to a document corpus 405 to generate a modified corpus 430 including text errors in the document corpus 405, wherein these errors are caused by different phoneme characteristics.

[0095] Text input 445 generally refers to a transcription of the input. Data from modified corpus 430 along with text input 445 can be input into context-aware correction model 450 for training purposes. In some embodiments, as part of pre-training phase 435, tags 440 that capture labeled training data can be generated. For example, tags 440 can be generated based on document corpus 405 and the output of context-aware correction model 450.

[0096] Some embodiments involve training a machine learning model to generate mistranscribed terms for uncommon terms in a plurality of pairs. For example, the context-aware correction model 450 may be trained to generate mistranscribed terms for uncommon terms. For example, uncommon terms may be identified in the document corpus 405, and the context-aware correction model 450 may be trained to generate mistranscribed terms for the identified uncommon terms. At runtime, when a potentially mistranscribed term is identified in speech as being transcribed, the potentially mistranscribed term may be compared to various mistranscribed terms to identify a correct uncommon term that may be used to replace the potentially mistranscribed term in the transcribed speech.

[0097] Some embodiments relate to adjusting a trained machine learning model based on an application of a computing device. For example, an uncommon term may be associated with a first set of incorrectly transcribed terms corresponding to a first application, and associated with a second set of incorrectly transcribed terms corresponding to a second application. For example, a particular uncommon term may be associated with a set of incorrectly transcribed terms from a music library, while the same uncommon term may be associated with a different set of incorrectly transcribed terms based on a web browser application. In some embodiments, the adjustment of the trained machine learning model involves generating incorrectly transcribed terms in multiple pairs based on one or more errors associated with the application. For example, in some music genres, uncommon terms may be pronounced in a way that may be a phonetically incorrect version of the uncommon term. For example, the uncommon term "tomato" may be pronounced as "tometo", "tamaato", and the trained machine learning model may identify "tometo", "tamaato" as incorrectly transcribed terms for "tomato".

[0098] In some embodiments, various models 415, 420, and 425 may be trained on a server and packaged into a device. In addition, for example, various models 415, 420, and 425 may be trained on a server, and updates may be provided to the device to train the context-aware correction model 450.

[0099] The generated incorrectly transcribed terms are stored in a local memory of the computing device. In some embodiments, the incorrectly transcribed terms are not transmitted outside the device (e.g., shared with a server or another device). In some embodiments, the incorrectly transcribed terms may be associated with an application. In such embodiments, the incorrectly transcribed terms cannot be shared between applications. For example, a first set of incorrectly transcribed terms associated with a first application is stored in a first memory associated with the first application, and a second set of incorrectly transcribed terms associated with a second application is stored in a second memory associated with the second application, and the first memory and the second memory are partitioned or firewalled from each other.

[0100] Generally speaking, data generation of mistranscribed terms and pairs of mistranscribed terms and uncommon terms, as well as training of machine learning models, can be performed on the user's device. This can bring huge computational advantages, especially when millions of users may generate billions of terms per hour.

[0101] The generated mis-transcribed terms extend the capabilities of existing input recognition systems. For example, the context-aware correction model 450 uses uncommon terms from the user's previous interactions with the computing device, and the total number of mis-transcribed terms that need to be considered for speech correction purposes is limited. This approach reduces both battery power and computing power, which is particularly significant on resource-constrained mobile devices.

[0102] In some embodiments, the generated mistranscribed terms can be applied to any upstream input recognition system. This approach ensures that the modified input recognition can be performed on the server or on the device according to the best resources available. For example, a hybrid approach can be applied, where the context-aware correction model 450 is applied in resource-constrained environments (e.g., mobile devices, low network bandwidth, etc.), while a server-based input recognition system using deep learning can be used when resources are less constrained (e.g., desktop computing devices, high network bandwidth, etc.).

[0103] Figure 5 5 is an example neural network 500 of a context-aware correction model for input recognition according to an example embodiment. As noted, user characteristics 505 may be used to determine a user profile 510. In some embodiments, user characteristics 505 may include one or more languages ​​associated with the user, dialects associated with the user, geographic or location information, etc. User-specific modulations 515 may be applied to layers of the neural network. For example, user-specific modulations 515 may include a personalized persistent API sequencer. Additionally, for example, a user may be prompted to speak a given sentence, and the user's voice characteristics may be inferred based on the spoken sentence. This personalization enables the input recognition system to be adapted to the user based on user-specific attributes.

[0104] In some embodiments, a pre-trained model with an accent modulation unit may be deployed on a user computing device. A bias and / or fine-tuning process may be performed based on user information such as contacts, location, etc. This user information is personalized, and the context-aware correction model is configured to be trained on the device, and any generated data is also stored locally on the device. For example, a user may be viewing a transcript, a medical prescription, a disease diagnosis, a financial document, or other forms of protected data, and the context-aware correction model may be configured to extract uncommon terms from such user activities and save them in the local memory of the device. Therefore, although user information is used, the data does not leave the user's computing device. This allows a private and personalized input recognition system to be provided to the user without having to use user history records and other forms of user interaction not provided by the user. Minimizing the use of user history records and / or logs can lead to significant improvements in the use of computing resources. In some embodiments, personal information and / or other forms of protected data may be encrypted on the device to further protect it.

[0105] The neural network 500 may be a deep neural network with an encoder-decoder architecture and several intermediate layers. In some embodiments, the neural network 500 may utilize an on-device neural machine translation (NMT) model. For example, an FNet model may be applied to a language task in which the self-attention layer of the encoder is replaced with a Fourier layer that applies a two-dimensional (2D) Fourier transform to the input. At run time, the input 520 may be an uncommon term. The encoder 525 may encode the input 520, which then passes through one or more layers of the neural network 500. User-specific modulation may provide user attributes to one or more layers. The decoder 530 may then determine the terms corresponding to the incorrect transcriptions of the uncommon terms provided as input 520. These incorrectly transcribed terms are generated as outputs 535 of the neural network 500. Although the encoder 525 and the decoder 530 are shown as a single block, the neural network 500 may also include multiple encoders and decoders.

[0106] Train machine learning models to generate inferences / predictions

[0107] Figure 6 A diagram 600 is shown illustrating a training phase 602 and an inference phase 604 of a trained machine learning model 632, according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to identify patterns in the training data and provide output inferences and / or predictions about (patterns in) the training data. The resulting trained machine learning algorithms may be referred to as trained machine learning models. For example, Figure 6 A training phase 602 is shown in which a machine learning algorithm 620 is trained on training data 610 to become a trained machine learning model 632. Then, during an inference phase 604, the trained machine learning model 632 may receive input data 630 and one or more inference / prediction requests 640 (possibly as part of the input data 630), and responsively provide one or more inferences and / or predictions 650 as output.

[0108] Thus, the trained machine learning model 632 may include one or more models of the machine learning algorithm 620. The machine learning algorithm 620 may include, but is not limited to, an artificial neural network (e.g., a convolutional neural network, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system as described herein). The machine learning algorithm 620 may be supervised or unsupervised, and may implement any suitable combination of online learning and offline learning.

[0109] In some examples, the machine learning algorithm 620 and / or the trained machine learning model 632 may be accelerated using an on-device coprocessor, such as a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and / or an application-specific integrated circuit (ASIC). Such an on-device coprocessor may be used to accelerate the machine learning algorithm 620 and / or the trained machine learning model 632. In some examples, the trained machine learning model 632 may be trained, resident, and executed to provide inferences on a particular computing device, and / or may otherwise make inferences for a particular computing device.

[0110] During the training phase 602, the machine learning algorithm 620 may be trained using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques by providing at least the training data 610 as training input. Unsupervised learning involves providing a portion (or all) of the training data 610 to the machine learning algorithm 620, and the machine learning algorithm 620 determining one or more output inferences based on the portion (or all) of the provided training data 610. Supervised learning involves providing a portion of the training data 610 to the machine learning algorithm 620, wherein the machine learning algorithm 620 determines one or more output inferences based on the portion of the provided training data 610, and accepting or correcting the output inferences based on correct results associated with the training data 610. In some examples, supervised learning of the machine learning algorithm 620 may be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm 620.

[0111] Semi-supervised learning involves obtaining the correct result for a portion (but not all) of the training data 610. During semi-supervised learning, supervised learning is used for a portion of the training data 610 that has the correct result, and unsupervised learning is used for a portion of the training data 610 that does not have the correct result. Reinforcement learning involves the machine learning algorithm 620 receiving a reward signal about a previous inference, where the reward signal can be a numerical value. During reinforcement learning, the machine learning algorithm 620 may output an inference and receive a reward signal in response, where the machine learning algorithm 620 is configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time. In some examples, the machine learning algorithm 620 and / or the trained machine learning model 632 can be trained using other machine learning techniques, including but not limited to incremental learning and curriculum learning.

[0112] In some examples, the machine learning algorithm 620 and / or the trained machine learning model 632 may use transfer learning techniques. For example, the transfer learning technique may involve the trained machine learning model 632 being pre-trained on a set of data and additionally trained using the training data 610. More specifically, the machine learning algorithm 620 may be pre-trained on data from one or more computing devices, and the resulting trained machine learning model is provided to a specific computing device, wherein the specific computing device is intended to execute the trained machine learning model during the inference phase 604. Then, during the training phase 602, the pre-trained machine learning model may be additionally trained using the training data 610, wherein the training data 610 may be derived from the kernel data and non-kernel data of the specific computing device. This further training of the machine learning algorithm 620 and / or the pre-trained machine learning model using the training data 610 of the specific computing device data may be performed using supervised learning or unsupervised learning. Once the machine learning algorithm 620 and / or the pre-trained machine learning model has been trained on at least the training data 610, the training phase 602 may be completed. The trained resulting machine learning model may be used as at least one of the trained machine learning models 632.

[0113] In particular, once the training phase 602 has been completed, the trained machine learning model 632 can be provided to the computing device (if not already on the computing device). The inference phase 604 can begin after the trained machine learning model 632 is provided to a particular computing device.

[0114] During the inference phase 604, the trained machine learning model 632 may receive input data 630 and generate and output one or more corresponding inferences and / or predictions 650 about the input data 630. Thus, the input data 630 may be used as an input to the trained machine learning model 632 for providing corresponding inferences and / or predictions 650 to kernel components and non-kernel components. For example, the trained machine learning model 632 may generate inferences and / or predictions 650 in response to one or more inference / prediction requests 640. In some examples, the trained machine learning model 632 may be executed by a portion of other software. For example, the trained machine learning model 632 may be executed by an inference or prediction daemon so that it is readily available upon request for providing inferences and / or predictions. The input data 630 may include data from a specific computing device executing the trained machine learning model 632 and / or input data from one or more computing devices other than the specific computing device.

[0115] The input data 630 may include uncommon terms. Other types of input data are also possible. Inference and / or prediction 650 may include one or more mis-transcribed terms that represent phonetically similar alternatives to pronounce uncommon terms. In some embodiments, inference and / or prediction 650 may include pairs of uncommon terms and mis-transcribed terms, wherein the associated confidence level indicates the phonetic similarity of the mis-transcribed terms and the uncommon terms. Inference and / or prediction 650 may include other output data generated by a trained machine learning model 632 operating on the input data 630 (and training data 610). In some examples, the trained machine learning model 632 may use the output inference and / or prediction 650 as input feedback 660. The trained machine learning model 632 may also rely on past inferences as input for generating new inferences.

[0116] Convolutional neural network 500 may be an example of machine learning algorithm 620. After training, the trained version of convolutional neural network 500 may be an example of trained machine learning model 632. In the method, an example of one or more inference / prediction requests 640 may be a request to predict one or more mis-transcribed terms corresponding to uncommon terms, and a corresponding example of inference and / or prediction 650 may be outputting the one or more mis-transcribed terms.

[0117] Sample Data Network

[0118] Figure 7 A distributed computing architecture 700 is depicted in accordance with an example embodiment. The distributed computing architecture 700 includes server devices 708, 710 configured to communicate with programmable devices 704a, 704b, 704c, 704d, 704e via a network 706. The network 706 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, an intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 706 may also correspond to a combination of one or more LANs, WANs, intranets, and / or the public Internet.

[0119] although Figure 7Only five programmable devices are shown, but the distributed application architecture can serve tens, hundreds, or thousands of programmable devices. In addition, programmable devices 704a, 704b, 704c, 704d, 704e (or any additional programmable devices) can be any kind of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mounted device (HMD), a network terminal, a mobile computing device, and the like. In some examples, such as shown in programmable devices 704a, 704b, 704c, 704e, the programmable device can be directly connected to the network 706. In other examples, such as shown in programmable device 704d, the programmable device can be indirectly connected to the network 706 via an associated computing device (such as programmable device 704c). In this example, programmable device 704c can act as an associated computing device for transmitting electronic communications between programmable device 704d and network 706. In other examples, such as shown in programmable device 704e, the computing device can be part of a vehicle (such as a car, truck, bus, boat or ship, airplane, etc.) and / or located inside the vehicle. In Figure 7 In other examples not shown, the programmable device may be connected to the network 706 either directly or indirectly.

[0120] The server devices 708, 710 may be configured to perform one or more services as requested by the programmable devices 704a-704e. For example, the server devices 708 and / or 710 may provide content to the programmable devices 704a-704e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data (such as compiled software), images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are also possible.

[0121] As another example, server devices 708 and / or 710 may provide programmable devices 704a-704e with access to software for database, search, computing, graphics, audio, video, web / internet utilization, and / or other functions. Many other examples of server devices are possible.

[0122] Computing device architecture

[0123] Figure 8 is a block diagram of an example computing device 800 according to an example embodiment. Specifically, Figure 8 The illustrated computing device 800 may be configured to perform at least one function of and / or related to the context-aware correction model, neural network, method 900 , and / or method 1000 as described herein.

[0124] The computing device 800 may include a user interface module 801, a network communication module 802, one or more processors 803, a data storage device 804, one or more cameras 818, one or more sensors 820 and a power system 822, all of which may be linked together via a system bus, network or other connection mechanism 805.

[0125] The user interface module 801 may be operable to send data to and / or receive data from an external user input / output device. For example, the user interface module 801 may be configured to send data to and / or receive data from a user input device such as a touch screen, a computer mouse, a keyboard, a keypad, a touch pad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 801 may also be configured to provide output to a user display device such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, whether known now or developed later. The user interface module 801 may also be configured to generate sound output using devices such as speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. The user interface module 801 may also be configured with one or more tactile devices that may generate tactile outputs such as vibrations and / or other outputs that may be detected by touch and / or physical contact with the computing device 800. In some examples, user interface module 801 may be used to provide a graphical user interface (GUI) for utilizing computing device 800 .

[0126] The network communication module 802 may include one or more devices that provide a wireless interface 807 and / or a wired interface 808 that may be configured to communicate via a network. The wireless interface 807 may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth® wireless communication module. TM Transceiver, Transceiver, Wi-Fi TM Transceiver, WiMAX TM Transceiver, LTE TM The wired interface 808 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or a similar transceiver that can be configured to communicate via a twisted pair, a coaxial cable, a fiber optic link, or a similar physical connection to a wired network.

[0127] In some examples, the network communication module 802 may be configured to provide reliable, protected and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., ensuring message delivery) may be provided, which may be part of a message header and / or message trailer (e.g., packet / message ordering information, encapsulation header and / or encapsulation trailer, size / time information, and transmission verification information (such as cyclic redundancy check (CRC) and / or parity check value)). One or more cryptographic protocols and / or algorithms may be used to secure (e.g., encoded or encrypted) and / or decrypt / decode communications, such as, but not limited to, the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA) algorithm, the Diffie-Hellman algorithm, a secure socket protocol such as a secure socket layer (SSL) or a transport layer security (TLS), and / or a digital signature algorithm (DSA). In addition to those protocols and / or algorithms listed herein, other cryptographic protocols and / or algorithms may be used to protect communications (and then decrypt / decode them).

[0128] The one or more processors 803 may include one or more general purpose processors and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). The one or more processors 803 may be configured to execute computer readable instructions 806 contained in the data storage device 804 and / or other instructions as described herein.

[0129] The data storage device 804 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 803. The one or more computer-readable storage media may include volatile and / or non-volatile storage components that may be fully or partially integrated with at least one of the one or more processors 803, such as optical, magnetic, organic or other memory or disk storage devices. In some examples, the data storage device 804 may be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disk storage unit), while in other examples, the data storage device 804 may be implemented using two or more physical devices.

[0130] Data storage 804 may include computer-readable instructions 806 and possibly additional data. In some examples, data storage 804 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, data storage 804 may include storage for trained neural network models 812 (e.g., models of trained convolutional neural networks such as convolutional neural network 140). Specifically, in these examples, computer-readable instructions 806 may include instructions that, when executed by one or more processors 803, enable computing device 800 to provide some or all of the functionality of trained neural network model 812.

[0131] In some examples, computing device 800 may include camera 818. Camera 818 may include one or more image capture devices, such as a still camera and / or a video camera, that are configured to capture light and record the captured light in one or more images; that is, camera 818 may generate images of the captured light. The one or more images may be one or more still images and / or one or more images used in video imagery. Camera 818 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.

[0132] In some examples, computing device 800 may include one or more sensors 820. Sensors 820 may be configured to measure conditions within computing device 800 and / or conditions in the environment of computing device 800 and provide data about these conditions. For example, sensors 820 may include one or more of the following: (i) sensors for obtaining data about computing device 800, such as, but not limited to, a thermometer for measuring the temperature of computing device 800, a battery sensor for measuring the charge of one or more batteries of power system 822, and / or other sensors for measuring conditions of computing device 800; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcode (e.g., quick response (QR) code) readers, and laser trackers, where identification sensors may be configured to read identifiers (such as RFID tags, barcodes, QR codes) and / or other devices and / or objects that are configured to be read and provide at least identification information; (iii) sensors for measuring the location and / or location of computing device 800; (iv) an environmental sensor for obtaining data indicative of the environment of the computing device 800, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (v) a force sensor for measuring one or more forces (e.g., inertial forces and / or gravity) acting on the computing device 800, such as, but not limited to, one or more sensors measuring force, torque, ground force, friction in one or more dimensions, and / or a ZMP sensor that identifies a zero moment point (ZMP) and / or the location of the ZMP. Many other examples of sensors 820 are also possible.

[0133] The power system 822 may include one or more batteries 824 and / or one or more external power interfaces 826 for providing power to the computing device 800. Each of the one or more batteries 824 may act as a storage power source for the computing device 800 when electrically coupled to the computing device 800. The one or more batteries 824 of the power system 822 may be configured to be portable. Some or all of the one or more batteries 824 may be easily removable from the computing device 800. In other examples, some or all of the one or more batteries 824 may be located inside the computing device 800 and therefore may not be easily removable from the computing device 800. Some or all of the one or more batteries 824 may be rechargeable. For example, a rechargeable battery may be charged via a wired connection between the battery and another power supply device (such as by being located outside the computing device 800 and connected to the computing device 800 via one or more external power interfaces). In other examples, some or all of the one or more batteries 824 may be non-rechargeable batteries.

[0134] The one or more external power interfaces 826 of the power system 822 may include one or more wired power interfaces, such as USB cables and / or power cords, that implement a wired power connection to one or more power supply devices located outside the computing device 800. The one or more external power interfaces 826 may include one or more wireless power interfaces, such as Qi wireless chargers, that implement wireless power connections to one or more external power supply devices. Once a power connection to an external power source is established using the one or more external power interfaces 826, the computing device 800 can draw power from the external power source through the established power connection. In some examples, the power system 822 may include associated sensors, such as battery sensors or other types of power sensors associated with one or more batteries.

[0135] Example methods of operation

[0136] Fig. 9 9 is a flow diagram of a method 900 according to an example embodiment. The method 900 may be performed by a computing device, such as the computing device 800. The method 900 may begin at block 910, where the method involves receiving, by an on-device system running on the computing device, input from a user during interaction with the computing device.

[0137] At block 920 , the method further involves receiving a transcription of the input from the input recognition model.

[0138] The method also involves identifying, by the on-device system, a candidate term for replacement in the input transcription at block 930. The candidate term may have been transcribed incorrectly.

[0139] At box 940, the method further involves accessing, by the on-device system and based on the candidate terms, a plurality of pairs of mis-transcribed terms and uncommon terms, wherein the uncommon terms may be mis-transcribed and wherein the mis-transcribed terms are incorrect versions of the uncommon terms that have been generated by the machine learning model.

[0140] At block 950 , the method further involves replacing, by the on-device system and based on the plurality of pairs of the incorrectly transcribed terms and the uncommon terms, the candidate term in the input transcription with the uncommon term.

[0141] In some embodiments, replacement of the candidate term involves comparing, by the on-device system, the candidate term to one or more mistranscribed terms from the plurality of pairs. Such embodiments also involve determining, based on whether the candidate term matches an mistranscribed term in the one or more mistranscribed terms, whether to replace the candidate term with a corresponding uncommon term paired with the matching mistranscribed term.

[0142] In some embodiments, the method involves identifying, by the on-device system, additional terms to be replaced in the input transcription. Such embodiments also involve determining that the additional terms do not match one or more incorrectly transcribed terms. Such embodiments further involve maintaining the additional terms in the input transcription.

[0143] In some embodiments, replacement of a candidate term involves determining that the candidate term matches a particular mistranscribed term of a particular pair of a plurality of pairs, wherein the particular pair is associated with a particular confidence level indicating a similarity of the particular mistranscribed term and a particular uncommon term paired with the particular mistranscribed term. The confidence level may be determined as a numerical value (e.g., between 0 and 1, on a scale of 1 to 10, as a percentage value, etc.). In some embodiments, the confidence level may be determined as a quality assessment, such as "high," "medium," or "low." Additional intermediate values ​​may be determined as "medium-high," "medium-low," "very high," "very low," etc.

[0144] Some embodiments also involve determining whether a particular confidence level exceeds a threshold. Generally speaking, the threshold may vary from one user to another, from one application to another, etc. In addition, for example, the threshold may vary based on the user's geographic location and / or the language used. For example, the user may be in a country or region where a particular language or dialect is spoken, and the user may adjust the voice to suit local customs.

[0145] In some embodiments, replacement of the candidate term further involves determining that a particular confidence level exceeds a threshold.Such embodiments also involve replacing the candidate term with a particular uncommon term.

[0146] Some embodiments involve storing the plurality of pairs in a local repository of a computing device.Such embodiments also involve limiting access to the contents of the local repository to within the computing device.

[0147] Some embodiments involve training a machine learning model to generate mistranscribed terms for uncommon terms in a plurality of pairs.

[0148] In some embodiments, training of the machine learning model further involves training the machine learning model to determine a corresponding confidence level for each of a plurality of pairs, wherein a given confidence level for a given pair including a given incorrectly transcribed term and a given uncommon term indicates a similarity of the given incorrectly transcribed term to the given uncommon term.

[0149] In some embodiments, the training of the machine learning model further includes receiving a corpus of web documents. Such embodiments also relate to synthetically simulating one or more errors based on the corpus of web documents. In some embodiments, the synthetic simulation of one or more errors is based on a text-to-speech model utilizing a noisy channel simulator. In some embodiments, the synthetic simulation of one or more errors is based on a grapheme-to-phoneme conversion model configured to generate a pronunciation of a word based on a text version of the word. In some embodiments, the synthetic simulation of one or more errors is based on a statistical phoneme model.

[0150] Some embodiments involve adjusting a trained machine learning model based on an application of a computing device. In some embodiments, the adjusting of the trained machine learning model involves generating a plurality of pairs of incorrectly transcribed terms based on one or more errors associated with the application.

[0151] Some embodiments involve receiving, by a computing device, a second input from a user during a second interaction with the computing device, wherein a second transcription of the second input includes a candidate term. Such embodiments involve comparing the candidate term to one or more second incorrectly transcribed terms, wherein the one or more second incorrectly transcribed terms are different from the one or more incorrectly transcribed terms. Such embodiments also involve replacing, by the on-device system, the candidate term in the transcription of the input with a second uncommon term based on the comparison, wherein the second uncommon term is paired with the second incorrectly transcribed term in the one or more second incorrectly transcribed terms.

[0152] Some embodiments relate to comprehensively simulating uncommon terms in a plurality of pairs. For example, uncommon terms can be generated based on aggregate statistics of uncommon terms that are commonly recognizable or known. Additionally, for example, uncommon terms can be generated based on aggregate statistics of commonly mistranscribed terms (e.g., based on speech-to-text transcription, autocorrection processes, etc.). In some embodiments, the comprehensive simulation of uncommon terms can be based on a trained machine learning model, e.g., as described in the context of generating mistranscribed terms.

[0153] In some embodiments, in one or more past interactions of the user with the computing device, the on-device system observed uncommon terms in the plurality of pairs.

[0154] In some embodiments, one or more past interactions of the user with the computing device involved interactions with an application of the computing device.

[0155] In some embodiments, one or more past interactions of the user with the computing device involved voice interactions.The uncommon terms in the plurality of pairs are based on user confirmation of the terms based on the transcription of the voice interactions.

[0156] In some embodiments, one or more past interactions of the user with the computing device involved interactions with a text editor.The uncommon terms in the plurality of pairs are based on user confirmation of the text terms in the text editor.

[0157] In some embodiments, the computing device includes a viewer interface. One or more past interactions of the user with the computing device involved textual content provided by the viewer interface. Uncommon terms in the plurality of pairs appear in the textual content.

[0158] In some embodiments, one or more past interactions of a user with a computing device may be one of past interactions with a search application, a text messaging application, an email messaging application, an instant messaging application, a web browsing application, a media playback application, a telephony application, a video communication application, a gaming application, a mapping application, or a navigation application.

[0159] In some embodiments, the input from the user is based on user input from a keyboard. In such embodiments, the input may be received as one or more of a typing operation, a swiping operation, or a gesture operation. In some embodiments, the plurality of pairs of mis-transcribed terms and uncommon terms may be based on one or more contextual variations of the plurality of uncommon terms.

[0160] In some embodiments, the input from the user is voice-based user input.

[0161] In some embodiments, the input recognition model may be a cloud-based automatic input recognizer.

[0162] In some embodiments, interaction with a computing device involves interaction with an application of the computing device.

[0163] In some embodiments, the interaction with the computing device can be one of an interaction with a search application, a text messaging application, an email messaging application, an instant messaging application, a web browsing application, a media playback application, a telephony application, a video communication application, a gaming application, a mapping application, or a navigation application.

[0164] In some embodiments, the interaction with the computing device may be an interaction with a digital assistant associated with the computing device.

[0165] Fig.10 1 is a flow diagram of a method 1000 according to an example embodiment. The method 1000 may be performed by a computing device, such as the computing device 800. The method 1000 may begin at block 1010, where the method involves receiving, by an on-device system running on the computing device, input from a user during interaction with the computing device.

[0166] At block 1020 , the method further involves receiving a transcription of the input from the input recognition model.

[0167] At block 1030 , the method also involves identifying candidate terms for replacement in the input transcription.

[0168] At block 1040, the method further involves determining whether the candidate term matches the incorrectly transcribed term. For example, the method involves identifying, by the on-device system, a candidate term for replacement in the input transcription. The candidate term may have been incorrectly transcribed. The method further involves replacing, by the on-device system and based on multiple pairs of the incorrectly transcribed term and the uncommon term, the candidate term in the input transcription with the uncommon term.

[0169] In some embodiments, replacement of the candidate term involves comparing, by the on-device system, the candidate term to one or more mistranscribed terms from the plurality of pairs. Such embodiments also involve determining, based on whether the candidate term matches an mistranscribed term in the one or more mistranscribed terms, whether to replace the candidate term with a corresponding uncommon term paired with the matching mistranscribed term.

[0170] Upon determining that the candidate term does not match the incorrectly transcribed term, the method further involves, at block 1050 , not replacing the candidate term in the input transcription.

[0171] Upon determining that the candidate term matches the mis-transcribed term, the method further involves, at block 1060 , determining whether a confidence level for a pair including the matching mis-transcribed term and the corresponding paired uncommon term exceeds a threshold.

[0172] Upon determining that the confidence level of the pair including the matching incorrectly transcribed term and the corresponding paired uncommon term does not exceed the threshold, the method further involves, at block 1050 , not replacing the candidate term in the input transcription.

[0173] Upon determining that a confidence level for a pair including a matching mistranscribed term and a corresponding paired uncommon term exceeds a threshold, the method further involves, at block 1070 , replacing the candidate term with the corresponding paired uncommon term in the input transcription.

[0174] The present disclosure is not limited in terms of the specific embodiments described in this application, which are intended to be illustrative of various aspects. As will be appreciated by those skilled in the art, many modifications and variations may be made without departing from the spirit and scope of the present disclosure. In addition to those methods and devices listed herein, those skilled in the art will also understand from the foregoing description that the functionally equivalent methods and devices within the scope of the present disclosure. Such modifications and variations are intended to fall within the scope of the appended claims.

[0175] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the accompanying drawings, similar symbols generally identify similar components unless the context otherwise dictates. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the various aspects of the present disclosure, as generally described herein and as shown in the accompanying drawings, may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated herein.

[0176] With respect to any or all of the ladder diagrams, scenes, and flow charts in the accompanying drawings and as discussed herein, each frame and / or communication may represent information processing and / or information transmission according to an example embodiment. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, the functions described as frames, transmissions, communications, requests, responses, and / or messages may not be performed in the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. In addition, more or less frames and / or functions may be used together with any of the ladder diagrams, scenes, and flow charts discussed herein, and these ladder diagrams, scenes, and flow charts may be combined in part or in whole with each other.

[0177] The box representing information processing may correspond to a circuit that can be configured to perform a specific logical function of the method or technology described herein. Alternatively or additionally, the box representing information processing may correspond to a module, a fragment, or a portion of a program code (including related data). The program code may include one or more instructions that can be executed by a processor to implement a specific logical function or action in a method or technology. The program code and / or related data may be stored on any type of computer-readable medium (such as a storage device including a disk or hard drive or other storage medium).

[0178] Computer readable media may also include non-transitory computer readable media, such as non-transitory computer readable media that store data in a short period of time, such as register memory, processor cache, and random access memory (RAM). Computer readable media may also include non-transitory computer readable media that can store program code and / or data in a longer period of time, such as auxiliary or persistent long-term storage devices, such as, for example, read-only memory (ROM), optical disks or magnetic disks, compact disk read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. For example, computer readable media may be considered to be, for example, a computer readable storage medium or a tangible storage device.

[0179] In addition, the boxes representing one or more information transfers may correspond to information transfers between software modules and / or hardware modules in the same physical device. However, other information transfers may be between software modules and / or hardware modules in different physical devices.

[0180] With respect to embodiments including determining uncommon terms based on user interactions with a computing device, and / or using a machine learning model to determine mistranscribed terms, or interactions by a computing device with a cloud-based server, controls may be provided to allow the user to make choices about whether and when the systems, programs, or features described herein may be able to collect user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, user demographic information, user's current location, or other personal information) and whether to send content or communications from the server to the user. In addition, certain data may be processed in one or more ways prior to storage or use so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (such as to a city, zip code, or state level) in the case of location information obtained so that the user's specific location cannot be determined. Thus, the user may control what information is collected about the user, how the information is used, and what information is provided to the user.

[0181] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the claims.

Claims

1. A computer-implemented method comprising: receiving, by an on-device system executing on a computing device, input from a user during interaction with the computing device; receiving a transcription of the input from an input recognition model; identifying, by the on-device system, a candidate term for replacement in the transcription of the input, wherein the candidate term may have been incorrectly transcribed; accessing, by the on-device system and based on the candidate terms, a plurality of pairs of mis-transcribed terms and uncommon terms, wherein the uncommon terms are likely mis-transcribed, and wherein the mis-transcribed terms are incorrect versions of the uncommon terms, the mis-transcribed terms having been generated by a machine learning model; as well as The candidate term is replaced, by the on-device system and based on the plurality of pairs of incorrectly transcribed terms and uncommon terms, with an uncommon term in the transcription of the input.

2. The computer-implemented method of claim 1 , wherein replacing the candidate term further comprises: comparing, by the on-device system, the candidate term to one or more mistranscribed terms from the plurality of pairs; as well as A determination is made based on whether the candidate term matches an incorrectly transcribed term of the one or more incorrectly transcribed terms whether to replace the candidate term with an uncommon term that is paired with the matching incorrectly transcribed term.

3. The computer-implemented method of claim 2, further comprising: identifying, by the on-device system, additional terms to be replaced in the transcription of the input; determining that the additional term does not match the one or more incorrectly transcribed terms; and The additional terms are maintained in the transcription of the input.

4. The computer-implemented method of claim 2, wherein replacing the candidate term further comprises: determining that the candidate term matches a particular mistranscribed term of a particular pair of the plurality of pairs, wherein the particular pair is associated with a particular confidence level indicating a similarity of the particular mistranscribed term and a particular uncommon term paired with the particular mistranscribed term; and A determination is made as to whether the particular confidence level exceeds a threshold.

5. The computer-implemented method of claim 4, wherein replacing the candidate term further comprises: determining that the certain confidence level exceeds the threshold; as well as The candidate term is replaced with the specific uncommon term.

6. The computer-implemented method of claim 1 , further comprising: The plurality of pairs are stored in a local repository of the computing device.

7. The computer-implemented method of claim 6, further comprising: Access to the contents of the local repository is restricted to within the computing device.

8. The computer-implemented method of claim 1 , further comprising: The machine learning model is trained to generate the incorrectly transcribed terms for the uncommon terms in the plurality of pairs.

9. The computer-implemented method of claim 8, wherein training the machine learning model further comprises: The machine learning model is trained to determine a respective confidence level for each of the plurality of pairs, wherein a given confidence level for a given pair including a given mistranscribed term and a given uncommon term indicates a similarity of the given mistranscribed term to the given uncommon term.

10. The computer-implemented method of claim 8, wherein training the machine learning model further comprises: receiving a corpus of documents; as well as One or more errors are synthetically modeled based on the corpus of documents.

11. The computer-implemented method of claim 10, wherein the synthetic simulation of the one or more errors is based on a text-to-speech model utilizing a noisy channel simulator.

12. The computer-implemented method of claim 10, wherein the synthetic simulation of the one or more errors is based on a grapheme-to-phoneme conversion model configured to generate a pronunciation of a word based on a text version of the word.

13. The computer-implemented method of claim 10, wherein the comprehensive simulation of the one or more errors is based on a statistical phoneme model.

14. The computer-implemented method of claim 8, further comprising: The trained machine learning model is adjusted based on an application of the computing device.

15. The computer-implemented method of claim 14, wherein the adjusting of the trained machine learning model comprises generating the incorrectly transcribed terms in the plurality of pairs based on one or more errors associated with the application.

16. The computer-implemented method of claim 1, further comprising: receiving, by the computing device, a second input from the user during a second interaction with the computing device, wherein a second transcription of the second input includes the candidate term; comparing the candidate term to one or more second incorrectly transcribed terms, wherein the one or more second incorrectly transcribed terms are different from the one or more incorrectly transcribed terms; as well as The candidate term is replaced in the transcription of the input with a second uncommon term by the on-device system and based on the comparison of the candidate term with one or more second incorrectly transcribed terms, wherein the second uncommon term is paired with a second incorrectly transcribed term of the one or more second incorrectly transcribed terms.

17. The computer-implemented method of claim 1, further comprising synthetically modeling the uncommon terms in the plurality of pairs.

18. The computer-implemented method of claim 1, wherein the uncommon terms in the plurality of pairs are observed by the on-device system in one or more past interactions of the user with the computing device.

19. The computer-implemented method of claim 18, wherein the one or more past interactions of the user with the computing device include interactions with an application of the computing device.

20. The computer-implemented method of claim 18, wherein the one or more past interactions of the user with the computing device include voice interactions, and wherein the uncommon terms in the plurality of pairs are based on user confirmation of terms based on transcriptions of the voice interactions.

21. A computer-implemented method as described in claim 18, wherein the one or more past interactions of the user with the computing device include interactions with a text editor, and wherein the uncommon terms in the plurality of pairs are based on user confirmation of text terms in the text editor.

22. A computer-implemented method as described in claim 18, wherein the computing device includes a viewer interface, and wherein the one or more past interactions of the user with the computing device include text content provided by the viewer interface, and wherein the uncommon terms in the plurality of pairs appear in the text content.

23. A computing device comprising: one or more processors; and A data storage device having computer executable instructions stored thereon, the computer executable instructions, when executed by the one or more processors, causing the computing device to perform functions, the functions comprising: receiving, by an on-device system executing on a computing device, input from a user during interaction with the computing device; receiving a transcription of the input from an input recognition model; identifying, by the on-device system, a candidate term for replacement in the transcription of the input, wherein the candidate term may have been incorrectly transcribed; accessing, by the on-device system and based on the candidate terms, a plurality of pairs of mis-transcribed terms and uncommon terms, wherein the uncommon terms are likely mis-transcribed, and wherein the mis-transcribed terms are incorrect versions of the uncommon terms, the mis-transcribed terms having been generated by a machine learning model; and The candidate term is replaced, by the on-device system and based on the plurality of pairs of incorrectly transcribed terms and uncommon terms, with an uncommon term in the transcription of the input.

24. An article of manufacture comprising: One or more computer-readable media having computer-readable instructions stored thereon, the computer-readable instructions, when executed by one or more processors of a computing device, causing the computing device to perform functions, the functions comprising: receiving, by an on-device system executing on a computing device, input from a user during interaction with the computing device; receiving, by an input recognition model, a transcription of the input; identifying, by the on-device system, a candidate term for replacement in the transcription of the input, wherein the candidate term may have been incorrectly transcribed; accessing, by the on-device system and based on the candidate terms, a plurality of pairs of mis-transcribed terms and uncommon terms, wherein the uncommon terms are likely mis-transcribed, and wherein the mis-transcribed terms are incorrect versions of the uncommon terms, the mis-transcribed terms having been generated by a machine learning model; and The candidate term is replaced, by the on-device system and based on the plurality of pairs of incorrectly transcribed terms and uncommon terms, with an uncommon term in the transcription of the input.

Images