Contextual de-normalization for automatic speech recognition
By dynamically selecting the deregulator list using field metadata in the ASR system, the problems of high cost and long waiting time in the existing technology are solved, and personalized speech recognition results are deregulated, which improves the system efficiency and User experience.
Patent Information
- Application Number
- CN202411134203.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-11-16
- Filing Date
- 2019-09-03
- Publication Date
- 2024-11-12
AI Technical Summary
When converting normalized text into readable form, existing automatic speech recognition (ASR) systems require multiple denormalizer modules, resulting in high costs and increased waiting time, and the denormalization rules cannot be dynamically adjusted according to the specific needs of users.
By introducing context metadata in the ASR module, dynamically selecting and applying the denormalizer list, the voice input metadata related to the user context is used to modify the default list of the denormalizer, thereby generating personalized denormalized text.
It realizes the de-normalization of speech recognition results based on user situations without multiple ASR modules, which reduces system costs and waiting time, and improves the readability and user experience of output.
Smart Images

Figure CN118940723A_ABST
Abstract
Description
[0001] Description of the case
[0002] This application is a divisional application of Chinese invention patent application No. 201980058744.X, filed on September 3, 2019. Technical Field
[0003] The present disclosure relates to contextual denormalization for automatic speech recognition. Background Art
[0004] Automatic speech recognition (ASR) systems generate raw speech recognition results that include normalized text as a result of training these ASR systems on normalized text. Normalized text refers to text that is textualized, all letters are lowercase, and all punctuation is stripped. The ASR system then incorporates a set of denormalizers that are used to denormalize the raw speech recognition results into a form that is more readable to the user. For example, the set of denormalizers can transform the raw speech recognition results by textualizing spelled-out numbers (e.g., converting "one-hundred and seventy-seven" to "177"), capitalizing appropriate letters (e.g., converting "send a text to john" to "Send a text to John"), and adding punctuation (e.g., converting "thanks exclamation mark" to "thanks!"). Because a set of denormalizers corresponds to a series of denormalizer rules, where the first denormalizer starts with the raw speech recognition results and each subsequent denormalizer receives as input the output of the previous denormalizer, there is no single sequence of denormalization rules that is optimal for all ASR output use cases, since what is optimal depends in part on the state and preferences of the user. As a result, ASR systems typically rely on different ASR modules to transcribe speech depending on the application at hand. For example, one ASR module may be used when the user is dictating speech, while a separate ASR module may be used when the user is performing a voice search. The use of multiple ASR modules still does not solve customized denormalization for each user, and further adds increased cost and latency due to having to maintain and access multiple ASR modules in order to output speech recognition results. Summary of the invention
[0005] One aspect of the present disclosure provides a method for denormalizing raw speech recognition results. The method includes: receiving speech input from a user at data processing hardware of an automatic speech recognition (ASR) module. The ASR module includes a predefined set of denormalizers to be applied to denormalize normalized text generated by the ASR module. The method also includes: obtaining context metadata associated with the speech input by the data processing hardware; and generating raw speech recognition results corresponding to the speech input by the data processing hardware. The generated raw speech recognition results include normalized text. The method also includes: selecting a list of one or more denormalizers to be applied to the generated raw speech recognition results based on the context metadata associated with the speech input by the data processing hardware; and denormalizing the generated raw speech recognition results into denormalized text by the data processing hardware by sequentially applying the list of one or more denormalizers to the generated raw speech recognition results.
[0006] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, receiving voice input includes receiving audio data corresponding to the voice input from a user device associated with the user, and obtaining context metadata associated with the voice input includes receiving context metadata associated with the voice input from the user device when the audio data of the voice input is received. In some examples, obtaining context metadata associated with the voice input includes: receiving a user identifier associated with the user when the voice input is received; and using the user identifier to retrieve context metadata from memory hardware that communicates with data processing hardware. The context metadata may include at least one of the following: a speech recognition category associated with the voice input, an application running on a user device associated with the user, the location of the user when the voice input is received, a user identifier identifying the user, demographic information associated with the user, or a user preference for denormalizing text. In some implementations, selecting the list of one or more denormalizers includes modifying a default list of denormalizers from a predefined set of denormalizers by at least one of: removing one or more denormalizers from the default list of denormalizers or adding one or more denormalizers from the predefined set of denormalizers to the default list of denormalizers.
[0007] In some implementations, the method further includes: providing, by data processing hardware, context metadata associated with the speech input to a context server; and receiving, at the data processing hardware, a denormalization instruction from the context server, the context server being configured to generate, based on the context metadata, a denormalization instruction for denormalizing the generated raw speech recognition result into denormalized text. In these implementations, selecting a list of one or more denormalizers includes selecting a list of one or more denormalizers using the denormalization instruction received from the context server. In some examples, providing the context metadata associated with the speech input to the context server includes providing the context metadata associated with the speech input to the context server without providing the speech input or any audio data associated with the speech input to the context server.
[0008] In some examples, the denormalization instructions include one or more of the following instructions, including: an append denormalizer instruction that identifies a new denormalizer to be appended at the beginning of a default list of denormalizers, the default list of denormalizers including a portion of a predefined set of denormalizers to be applied sequentially to the generated raw speech recognition results in the absence of context data; an append denormalizer instruction that identifies a new denormalizer to be appended at the end of the default list of denormalizers; a remove denormalizer instruction that identifies an existing denormalizer to be removed from the default list of denormalizers; a replace denormalizer instruction that identifies a new denormalizer to replace an existing denormalizer from the default list of denormalizers; or an add denormalizer instruction that identifies a new denormalizer to be added to the default list of denormalizers and identifies an existing denormalizer from the default list of denormalizers that precedes or follows the new denormalizer. Alternatively, the denormalization instruction may include an overload instruction that overloads a default list of denormalizers with a new list of denormalizers. The new list of denormalizers may include a selected list of one or more denormalizers. In some implementations, the context server is configured to receive one or more customized denormalization rules specific to a client of the ASR module and compile a customized denormalizer based on the one or more customized denormalization rules specific to the client.
[0009] In some implementations, denormalizing the generated raw speech recognition result into denormalized text using a selected list of one or more denormalizers includes: retrieving each denormalizer in the selected list of one or more denormalizers from memory hardware using a corresponding key; and applying each denormalizer in the selected list of one or more denormalizers retrieved from the memory hardware to the generated raw speech recognition result in sequence. The memory hardware communicates with the data processing hardware. In some examples, the memory hardware is located on the ASR module. In other examples, the memory hardware includes a repository located away from the ASR module.
[0010] In some examples, the method further includes, after denormalizing the generated raw speech recognition result into denormalized text, transmitting the denormalized text from the ASR module to the client device. In these examples, the denormalized text, when received by the client device, is configured to cause the client device to display the denormalized text on a display screen of the client device. In some implementations, transmitting the denormalized text from the ASR module includes streaming the denormalized text to the client device in real time.
[0011] Another aspect of the present disclosure provides a system for denormalizing raw speech recognition results. The system includes data processing hardware of an automatic speech recognition (ASR) module and memory hardware of the ASR module in communication with the data processing hardware. The ASR module includes a predefined set of denormalizers to be applied to denormalize normalized text generated by the ASR module. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including: receiving speech input from a user, obtaining context metadata associated with the speech input, and generating raw speech recognition results corresponding to the speech input. The generated raw speech recognition results include normalized text. The operations also include: selecting a list of one or more denormalizers to be applied to the generated raw speech recognition results based on the context metadata associated with the speech input; and denormalizing the generated raw speech recognition results into denormalized text by sequentially applying the list of one or more denormalizers to the generated raw speech recognition results.
[0012] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, receiving a voice input includes receiving audio data corresponding to the voice input from a user device associated with a user, and obtaining context metadata associated with the voice input includes receiving context metadata associated with the voice input from the user device when the audio data of the voice input is received. In some examples, obtaining context metadata associated with the voice input includes: receiving a user identifier associated with the user when the voice input is received; and using the user identifier to retrieve context metadata from memory hardware. The context metadata may include at least one of: a speech recognition category associated with the voice input, an application running on a user device associated with the user, a location of the user when the voice input is received, a user identifier identifying the user, demographic information associated with the user, or a user preference for denormalizing text. In some implementations, selecting a list of one or more denormalizers includes modifying a default list of denormalizers from a predefined set of denormalizers by at least one of: removing one or more denormalizers from the default list of denormalizers or adding one or more denormalizers from a predefined set of denormalizers to the default list of denormalizers.
[0013] In some implementations, the operations further include: providing context metadata associated with the speech input to a context server; and receiving a denormalization instruction from the context server, the context server being configured to generate a denormalization instruction for denormalizing the generated raw speech recognition result into denormalized text based on the context metadata. In these implementations, selecting a list of one or more denormalizers includes selecting a list of one or more denormalizers using the denormalization instruction received from the context server. In some examples, providing the context metadata associated with the speech input to the context server includes providing the context metadata associated with the speech input to the context server without providing the speech input or any audio data associated with the speech input to the context server.
[0014] In some examples, the denormalization instructions include one or more of the following instructions, including: an append denormalizer instruction that identifies a new denormalizer to be appended at the beginning of a default list of denormalizers, the default list of denormalizers including a portion of a predefined set of denormalizers to be applied sequentially to the generated raw speech recognition results in the absence of context data; an append denormalizer instruction that identifies a new denormalizer to be appended at the end of the default list of denormalizers; a remove denormalizer instruction that identifies an existing denormalizer to be removed from the default list of denormalizers; a replace denormalizer instruction that identifies a new denormalizer to replace an existing denormalizer from the default list of denormalizers; or an add denormalizer instruction that identifies a new denormalizer to be added to the default list of denormalizers and identifies an existing denormalizer from the default list of denormalizers that precedes or follows the new denormalizer. Additionally or alternatively, the denormalization instruction may include an overload instruction that overloads a default list of denormalizers with a new list of denormalizers. The new list of denormalizers may include a selected list of one or more denormalizers. In some implementations, the context server is configured to receive one or more customized denormalization rules specific to a client of the ASR module and compile a customized denormalizer based on the one or more customized denormalization rules specific to the client.
[0015] In some examples, denormalizing the generated raw speech recognition result into denormalized text using a selected list of one or more denormalizers includes: retrieving each denormalizer in the selected list of one or more denormalizers from memory hardware using a corresponding key; and applying each denormalizer in the selected list of one or more denormalizers retrieved from the memory hardware to the generated raw speech recognition result in sequence. In some examples, the memory hardware is located on the ASR module. In other examples, the memory hardware includes a repository located away from the ASR module.
[0016] In some implementations, the operation further includes, after denormalizing the generated raw speech recognition result into denormalized text, transmitting the denormalized text from the ASR module to the client device. In these implementations, the denormalized text, when received by the client device, is configured to cause the client device to display the denormalized text on a display screen of the client device. In some examples, transmitting the denormalized text from the ASR module includes streaming the denormalized text to the client device in real time.
[0017] The details of one or more implementations of the present disclosure are set forth in the drawings and description that follow.Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 An example system for denormalizing raw speech recognition results from an automatic speech recognition (ASR) module is schematically illustrated.
[0019] Figure 2A and Figure 2B yes Figure 1 Schematic diagram of an ASR module denormalizing text based on context metadata from a user device and sending the denormalized text to the user device.
[0020] Figure 3 The context server is schematically illustrated to provide denormalization instructions to the denormalizer module for denormalizing the raw speech recognition results.
[0021] Figure 4A and Figure 4B It is a schematic diagram of a context server compiling a personalized denormalizer for a user based on personalized denormalization rules received from a user device associated with the user.
[0022] Figure 5 is a diagram of a sample output that receives denormalized text.
[0023] Figure 6 is a flow diagram of an example arrangement of operations of a method for denormalizing raw speech recognition results.
[0024] Figure 7 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.
[0025] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION
[0026] The use of denormalizers in automatic speech recognition (ASR) systems is paramount to outputting text that is grammatically correct and optimized for readability. The ASR module is trained on a corpus of text that is "normalized" in that the text is stripped of all punctuation, does not include capital letters, all numbers include literal text (e.g., "one-hundred and seventy-seven" instead of "177"), etc. As a result, the trained ASR module, when receiving speech during inference, initially generates raw speech recognition results (e.g., raw transcriptions) indicating the "normalized" text. In order to provide the user with a transcription optimized for reading, the ASR module employs a list of denormalizers to convert the raw speech recognition results into a more readable form, wherein the first denormalizer in the set of denormalizers receives the raw speech recognition results and each subsequent denormalizer receives as input the corresponding output of the previous denormalizer in the set of denormalizers. Each denormalizer is associated with a corresponding denormalization rule or denormalization model that denormalizes some aspect of the input of the text. For example, a capitalization denormalizer rule may convert the phrase "send a message to john" into "Send a message to John", a literalization denormalizer rule converts all literalized text (e.g., "sixty-five") into its written form (e.g., "65"), an implicit punctuation denormalizer rule adds punctuation (e.g., converts "thank you sir" into "thank you, sir"), an explicit punctuation denormalizer rule generates punctuation (e.g., converts "thanks exclamation mark" into "thanks!"), a literal denormalizer translates text into its corresponding characters, an ideographic denormalizer generates ideographic symbols (e.g., a smiley face emoticon) from its corresponding text (e.g., ":"), and an offensive word denormalizer masks offensive words by replacing one or more letters in the offensive word with stars, asterisks, and / or other characters.
[0027] The implementation of this article is directed to a single ASR module configured to dynamically apply a contextualized list of denormalizers to the original speech recognition results of the corresponding speech input based on the context of the speech input. Here, the ASR module can receive a speech input (e.g., a speech recognition request) from a user, generate an original speech recognition result (e.g., a normalized text) corresponding to the speech input, obtain context metadata associated with the speech input, and select a list of one or more denormalizers based on the context metadata associated with the speech input for denormalizing the generated original speech recognition results into denormalized text. The ASR module is configured to apply each denormalizer from a selected list of one or more denormalizers to the generated original speech recognition results in turn. Advantageously, a single ASR module including a predefined set of denormalizers can perform speech recognition tasks for an entire user group by using the context metadata associated with the speech input to select a suitable list of one or more denormalizers from a predefined set of denormalizers. Thus, a list of one or more denormalizers selected based on context metadata associated with a given speech input personalizes how raw speech recognition results corresponding to a given speech input are denormalized without requiring multiple ASR modules / engines.
[0028] The ASR module may receive context metadata associated with the voice input from the user when the voice input is received and / or retrieve at least a portion of the context metadata from another source after receiving the voice input. The context metadata associated with the voice input may include, but is not limited to, a voice recognition category associated with the voice input, an application running on a user device that is intended to consume denormalized text, a user identifier that identifies the user, the location of the user when the voice input is received, demographic information associated with the user, and / or a user preference for denormalizing the text. In some scenarios, the context metadata may include one or more personalized denormalization rules provided by the user for immediate application by the ASR module in order to denormalize the original voice recognition results. The ASR module includes an internal memory that stores a predefined set of denormalizers, and the ASR module can access the appropriate denormalizer from it based on the context metadata for denormalizing the original voice recognition results. For example, the internal memory may include a data storage device that maintains a key-value mapping of a predefined set of denormalizers, and the ASR module can retrieve any given denormalizer (i.e., value) from the data register by providing a corresponding key. Optionally, the ASR module may access a separate repository storing less frequently used denormalizers and retrieve those denormalizers from the repository on demand, thereby alleviating storage requirements at the ASR module. Additionally or alternatively, the ASR module may receive denormalization instructions that specify custom / personalized denormalizers for the ASR module to retrieve from other sources.
[0029] In some examples, the ASR module is configured to use a default list of denormalizers to denormalize the original speech recognition result whenever the context metadata associated with the corresponding speech input does not exist. Therefore, when the context metadata associated with the corresponding speech input is available, the ASR module is configured to modify the default list of denormalizers based on the context metadata. Specifically, the ASR module can forward the context metadata to the context server, which is configured to generate a denormalization instruction for denormalizing the generated original speech recognition result into a denormalized text based on the context metadata and provide the denormalization instruction to the ASR module. The ASR module can forward the context metadata to the context server without providing the speech input or any audio data associated with the speech input to the context server. Since the context server only has to process the context metadata to generate the denormalization instruction, the ASR module can generate the original speech recognition result while the context server concurrently generates the denormalization instruction and provides it to the ASR module, without causing any increased waiting time. Thereafter, the ASR module uses the denormalization instruction to select a list of one or more denormalizers by modifying the default list of denormalizers. For example, the denormalization instructions may instruct the ASR module to modify a default list of denormalizers by at least one of: removing one or more denormalizers from the default list of denormalizers, adding one or more new denormalizers to the default list of denormalizers, or specifying an order in which each denormalizer in the modified list of denormalizers is to be applied to denormalize the generated raw speech recognition results.
[0030] In some scenarios, the denormalization instructions include reload instructions to reload the default list of denormalizers with the new list of denormalizers. In these scenarios, the context server does not need to know the default list of denormalizers and can identify each denormalizer in the new list of denormalizers from a predefined set of denormalizers. Regardless of how the context server modifies or reloads the default list of denormalizers, the denormalization instructions only need to include a key for the ASR module to use to retrieve / retrieve each corresponding denormalizer (e.g., value) from the predefined set of denormalizers stored in the ASR module's data register. In some implementations, the reload instructions reload the default list of denormalizers with one or more personalized denormalizers that are specific to a given user and are not included in the predefined set of denormalizers. In these implementations, the denormalization instructions may include actual data corresponding to the personalized denormalizers for the ASR module to apply on the fly, or the denormalization instructions may include corresponding keys for the ASR module to use to retrieve the personalized denormalizers from a personalized denormalizer repository.
[0031] refer to Figure 1 In some implementations, the system 100 includes a user device 200 associated with a user 10, and the user 10 can communicate with a remote system 140 via a network 130. The remote system 140 can be a distributed system (e.g., a cloud computing environment) with scalable / elastic resources 142. The resources 142 include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). In some implementations, the remote system 140 executes an automatic speech recognition (ASR) module 300, 300a, which is configured to receive audio data 103 corresponding to a speech input 104 from the user device 200, generate a raw speech recognition result 312 corresponding to the speech input 104, and denormalize the generated raw speech recognition result 312 into a denormalized text 322. As used herein, the raw speech recognition result 312 corresponds to normalized text converted from the audio data 103 corresponding to the speech input 104, wherein the text is textualized and stripped of all punctuation and capitalized. Conversely, the denormalized text 322 includes a conversion of the raw speech recognition result 312 into a form that is more readable to the user 10. For example, denormalizing the raw speech recognition result 312 into the denormalized text 322 may include textualizing spelled-out numbers (e.g., converting "one-hundred and seventy-seven" to "177"), capitalizing appropriate letters (e.g., converting "send a text to john" to "Send a text to John"), and adding punctuation (e.g., converting "thanks exclamation mark" to "thanks!").
[0032] The user device 200 includes data processing hardware 204 and memory hardware 206. The user device 200 may include an audio capture device (e.g., a microphone) for capturing speech input 104 from the user 10 and converting it into audio data 103 (e.g., an electrical signal). In some implementations, the data processing hardware 204 is configured to execute an ASR module 300, 300b that replaces or substitutes the ASR module 300a executed on the remote system 140. The user device 200 can be any computing device capable of communicating with the communication device 300 via the network 130. The user device 200 includes, but is not limited to, desktop computing devices and mobile computing devices such as laptops, tablets, smart phones, and wearable computing devices (e.g., headphones and / or watches).
[0033] In the example shown, the user 10 may provide a speech input 104 to the user device 200 by speaking, and the user device 200 is capable of capturing the speech input 104 and converting it into audio data 103. The user device 200 may then transmit an ASR request 102 including the audio data 103 and the context metadata 110 associated with the speech input 104 to the ASR module 300a executed on the remote system 140. Here, the ASR request 102 requests the ASR 300 to annotate the speech input 104 into corresponding denormalized text 322 based on the context metadata 110 associated with the speech input 104. The user device 200 may optionally execute the ASR module 300b to annotate the speech input 104 into the corresponding denormalized text 322. For example, when network communication is interrupted or unavailable, the user device 200 may execute the ASR module 300b locally to annotate the captured speech input 104 into the corresponding denormalized text 322.
[0034] The ASR module 300 includes a recognizer 310, a denormalizer module 320, a data storage device 340, and a context server 400. Although the context server 400 is shown as a component of the ASR module 300, the context server 400 may be a separate component in communication with the ASR module 300 in other examples without departing from the scope of the present disclosure. The ASR module 300 is configured to receive the ASR request 102 and provide the audio data 103 corresponding to the speech input 104 to the recognizer 310 and forward the context metadata 110 to the context server 400. Here, the recognizer 310 is configured to generate a raw speech recognition result 312 (e.g., normalized text) corresponding to the speech input 104, while at the same time, the context server 400 is configured to generate a denormalization instruction 410 based on the context metadata 110 for denormalizing the raw speech recognition result 312. Therefore, since neither the speech input 104 nor the audio data 103 associated with the speech input 104 is provided to the context server 400, the context server 400 only has to process the context metadata 110 to generate the denormalization instructions 410. As a result, the ASR module 300 is able to generate the denormalization instructions 410 via the context server 400 and simultaneously generate the original speech recognition results 312 via the recognizer 310 without incurring increased latency.
[0035] The data storage 340 (e.g., memory hardware) of the ASR module 300 stores a predefined set of denormalizers 350, each of which corresponds to a respective denormalization rule or denormalization model for denormalizing some aspect of the input of the normalized text (e.g., the raw speech recognition results 312) received at the denormalizer module 320. For example, the predefined set of denormalizers 350 may include, but is not limited to, at least one of a capitalization denormalizer, a literalization denormalizer, an implicit punctuation denormalizer, an explicit punctuation denormalizer, a literalization denormalizer, a semantic denormalizer, or an offensive word denormalizer.
[0036] The denormalizer module 320 is configured to select a list of one or more normalizers 350 to be applied to the raw speech recognition results 312 based on the context metadata 110 associated with the speech input 104. More specifically, the denormalizer module 320 uses the denormalization instructions 410 generated by the context server 400 to select the list of one or more denormalizers 350 to be applied to denormalize the raw speech recognition results 312. In some examples, the denormalization instructions 410 identify the list of one or more denormalizers 350 for selection by the denormalizer module 320 by retrieving the corresponding denormalizer 350 from a predefined set of denormalizers 350 stored in the data storage device 340. In these examples, the instructions 410 identify each denormalizer and provide a corresponding key 344 ( Figure 3 ), the denormalizer module 320 uses the key to retrieve the corresponding denormalizer 350 from the data storage device 340. In other examples, the denormalization instruction 410 identifies other denormalizers, such as a customized / personalized denormalizer 450 (compiled on the fly for a specific user 10) Figure 4A and Figure 4B ) or infrequently used denormalizers that are not among the predefined set of denormalizers 350 within the data storage 340. In some implementations, the personalized denormalizer 450 associated with a specific user 10 is encrypted, and the denormalizer module 320 can only decrypt the personalized denormalizer when the user 10 provides an appropriate decryption key.
[0037] The context metadata 110 obtained by the context server 400 includes at least one of the following: a speech recognition category associated with the speech input 104, an application running on a user device 200 associated with the user 10, the location of the user 10 when the speech input 104 is received by the ASR module 300, a user identifier identifying the user, demographic information associated with the user 10, or a user preference for denormalizing text.
[0038] The speech recognition category associated with the speech input 104 can indicate whether the speech input 104 is to be converted into text for dictation or whether the speech input 104 is provided as a voice search or command for execution by the user device 200. In the latter case, the ASR module 300 converts the speech input 104 into a denormalized text 322 to be displayed on the user device 200 as an annotation of the voice search or command specified by the speech input 104. The speech recognition category associated with the speech input 104 can include other categories, such as but not limited to, transcribing the speech input 104 as part of a conversation, an audio / video conference, or a voice mail record. As will become apparent, the speech recognition category associated with the speech input 104 can determine which denormalizers 350 the denormalizer module 320 selects for denormalizing the original speech recognition result 312. In other words, the denormalizer module 320 can denormalize the same original speech recognition result 312 differently based on the speech recognition category associated with the corresponding speech input 104.
[0039] When the context metadata 110 includes an identifier of an application running on a user device 200 associated with the user 10, the context server 400 may be able to identify the speech recognition category associated with the speech input. For example, a dictation application currently open and running on the user device 200 can indicate that the user 10 is dictating the speech input 104. The context metadata 110 including the location of the user 10 when the speech input 104 is received by the ASR module 300 can be used by the context server 400 to identify the explicit language and / or dialect spoken at the location. For example, a literal translation denormalizer can be applied when the user 10 is located in Japan to convert the text into corresponding characters, but when the user 10 is located elsewhere, a different literal translation denormalizer may be applied or no literal translation denormalizer may be applied. The location within the context metadata 110 may include a geographic area, so that the exact geographic location of the user 10 cannot be determined. In addition, the user must explicitly agree to provide location information and has the ability to opt out of sharing location information at any time.
[0040] When the ASR request 102 provides context metadata 110 including a user identifier identifying a user, the context server 400 may use the user identifier to obtain additional context metadata 110 from one or more context sources 190. For example, the context server 400 may use the user identifier to retrieve the context metadata 110 from a user profile 192 associated with the user 10. The user profile 192 may include demographic information associated with the user 10, such as age, gender, residence, spoken language, education, occupation, etc. The user 10 voluntarily provides and agrees to store the demographic information for use by the ASR module 300 and has the option to remove the demographic information at any time. In some examples, as an alternative or in addition to the location of the user 10, the context server 400 identifies the dominant language and / or dialect spoken by the user 10 by analyzing the demographic information within the user profile 192 associated with the user 10. The user profile 192 may also include user preferences for denormalizing text. For example, the denormalizer module 320 may apply an offensive word denormalizer by default to mask offensive words in the text, but a user preference for causing the text to be denormalized may explicitly indicate that offensive words should not be masked. Thus, the context server 400 may specify in the denormalization instructions 410 that the denormalizer module 320 should not apply the offensive word denormalizer when denormalizing the raw speech recognition results 312 for this particular user 10. Thus, the context server 400 may receive some or all of the context metadata 110 upon receiving the ASR request 102 and may retrieve other context metadata 110 from the context source 190 by accessing the user profile 192 associated with the user identifier identifying the user 10.
[0041] After denormalizing the raw speech recognition results 312, the denormalizer module 320 can provide the denormalized text 322 to an output 500 that is configured to display and / or consume the denormalized text 322 in some other manner. Figure 5, the output 500 may include the user device 200 and / or one or more other user devices capable of presenting the denormalized text 322 for reading by the user. The output 500 may additionally or alternatively include one or more applications / services executed on the remote system 140. For example, the speech input 104 may include input to a message board service executed on the remote system 140, which the user device may access to view the denormalized text 322 corresponding to the speech input 104. The output 500 may also include a transcription service that provides denormalized text 322 corresponding to the transcribed speech to one or more speakers in the telecommunications conference session. Additionally or alternatively, the output 500 may include other systems / modules that consume the denormalized text 322 for purposes other than human readability. For example, the output 500 may include a text-to-speech system that receives the denormalized text 322 and generates synthesized speech. Here, the denormalized text 322 may include prosody and / or pronunciation tags from the corresponding denormalizer for the TTS system to apply when generating the synthesized speech. In another example, the output 500 may include a natural language understanding (NLU) module that uses the denormalized text 322 to understand what the user wants. In some examples, the denormalized text 322 consumed by the output 500 for non-human purposes (e.g., TTS systems and NLU modules) is not denormalized for readability, but is instead denormalized / normalized in a manner more suitable for processing by the corresponding output 500.
[0042] Figure 2A and Figure 2B An exemplary user device 200 is shown sending an ASR request 102 to an ASR module 300 and receiving denormalized text 322 from the ASR module 300. The user device 200 includes a screen 212 and executes a graphical user interface 214 for display on the screen 212. The user device 200 also executes one or more software applications 210, 210a-d. The software application 210 can refer to computer software that, when executed by a computing device, causes the computing device to perform a task. In some examples, the software application 210 can be referred to as an "application", "app" or "program". Example software applications 210 include, but are not limited to, voice communication applications 210a, media streaming applications 210b, messaging applications 210c, calendar applications 210d, word processing applications, spreadsheet applications, social networking applications, and games.
[0043] Figure 2AThe user device 200 is shown currently executing a calendar application 210d (represented by a solid line) and sending an ASR request 102 to the ASR module 300 at time 1, the ASR request 102 including audio data 103 and contextual metadata 110 associated with a speech input 104. Here, the speech input 104 corresponds to a voice search directed to the calendar application 210d to determine when to record a meeting time for a topic (discuss new line of tools) with a given person (Skip Fikany) on a given day (tomorrow). Among other types of metadata, the contextual metadata 110 associated with the speech input 104 and provided in the ASR request 102 includes an application identifier that identifies that the calendar application 210d is currently executing on the user device 200 and / or that the speech input 104 is directed to the calendar application 210d.
[0044] At time 2, the recognizer 310 generates a raw speech recognition result 312 corresponding to the speech input 104, and at time 3, the context metadata 110 is forwarded to the context server 400. Times 2 and 3 may occur concurrently or simultaneously with time 3 occurring before or after time 2. The raw speech recognition result 312 may include normalized text that narrates "when is my meeting tomorrow with skip fikany to discuss the new line of tools". Although the normalized text can be used by the calendar application 210d to perform a search to determine the time of the meeting, the normalized text is not optimized by the user for readability. Using the context metadata 110 including the application identifier identifying the calendar application 210d, the context server 400 is able to identify that the speech recognition category associated with the speech input 104 indicates that the speech input 104 is provided as a voice search for the user device 200 to perform by accessing the calendar application 210d. Thus, at time 4 , the context server 400 generates and provides the denormalization instructions 410 to the denormalizer module 320 based on the context metadata 110 indicating that a voice search application (eg, the calendar application 210 d ) is currently executing on the user device 200 .
[0045] The denormalizer module 320 uses the denormalization instructions 410 to select a list of one or more denormalizers 350 to be applied to the generated raw speech recognition results 312. For example, the denormalization instructions 410 may specify a list of one or more denormalizers 350 and the denormalizer module 320 may select one or more normalizers 350 from a predefined set of denormalizers 350 within the data storage device 340. Since the context metadata 110 indicates that the speech input 104 is associated with a voice search application (e.g., the calendar application 210d), the context server 400 will generate the denormalization instructions 410 that ensure that the denormalizer module 320 will not apply an explicit punctuation denormalizer because none of the words in the speech input 104 for voice search are likely to include explicit punctuation words. Thus, in this example, the denormalization instructions 410 will instruct the denormalizer module 320 to remove the explicit punctuation denormalizer from the default list of denormalizers, or when there is no default list or the default list does not include an explicit punctuation denormalizer, the denormalization instructions 410 will not instruct the denormalizer module 320 to include an explicit punctuation denormalizer. The list of one or more denormalizers indicates the order in which the denormalizers should be applied such that the first denormalizer is applied to the raw speech recognition result 312 and each subsequent denormalizer receives the output from the previous denormalizer as input. At time 5, the denormalizer module 320 denormalizes the raw speech recognition result 312 into denormalized text 322 by sequentially applying the list of one or more denormalizers 350 to the generated raw speech recognition result 312.
[0046] At time 6, the ASR module 300 provides the denormalized text 322 corresponding to the speech input 104 to the user device 200, and the user device 200 displays the denormalized text 322 in the voice search / command window 216 displayed in the graphical user interface 214. Here, the denormalized text 322 only annotates the speech input 104 corresponding to the voice search directed to the calendar application 210d. In the example shown, the denormalized text 322 is denormalized by at least a capital denormalizer and an implicit denormalizer. For example, the denormalizer module 320 is applied with a capital denormalizer to capitalize the first letter "W" that starts the voice search, the first letter "S" of the personal name (Skip), and the first letter "F" of the personal surname (Fikany). Because the speech input 104 is phrased as a question, the implicit denormalizer adds a question mark "?" at the end of the voice search. Notably, because the denormalizer module 320 does not apply an explicit punctuation denormalizer to the raw speech recognition result 312, the word "new line" in the raw speech recognition result 312 is included in the denormalized text 322 and is not converted into a spoken text that would incorrectly omit the word "new line" and instead begin a line break n / after the word "discuss" beginning with the word "of tools." In some examples, the ASR module 300 streams the denormalized text 322 to the user device 200 (or other output 500) in real time so that the denormalized text 322 is displayed while the user 10 is uttering speech.
[0047] Figure 2B The user device 200 is shown currently executing a messaging application 210c (represented by a solid line) and sending an ASR request 102 to the ASR module 300 at time 1, the ASR request 102 including audio data 103 associated with a speech input 104 and contextual metadata 110. Here, the speech input 104 corresponds to spoken speech directed to the messaging application 210c for inclusion in an electronic message. Among other types of metadata, the contextual metadata 110 associated with the speech input 104 and provided in the ASR request 102 includes an application identifier that identifies that the messaging application 210c is currently executing on the user device 200 and / or that the speech input 104 is provided to the messaging application 210c.
[0048] At time 2, the recognizer 310 generates a raw speech recognition result 312 corresponding to the speech input 104, and at time 3, forwards the context metadata 110 to the context server 400. Times 2 and 3 may occur concurrently or simultaneously with time 3 occurring before or after time 2. The raw speech recognition result 312 may include normalized text that narrates "dear carrie commanew line thank you for the cheesecake exclamation point." Using the context metadata 110 including the application identifier identifying the messaging application 210c, the context server 400 is able to discern that the speech recognition category associated with the speech input 104 indicates that the speech input 104 is provided as spoken speech to be converted into text for the messaging application 210c executing on the user device 200. Thus, at time 4, the context server 400 generates a denormalization instruction 410 based on the context metadata 110 indicating that the dictation application (e.g., the messaging application 210c) is currently executing on the user device 200 and provides the denormalization instruction 410 to the denormalizer module 320. Here, the context server 400 will generate the denormalization instruction 410, which ensures that the denormalizer module 320 will apply an explicit punctuation denormalizer because the speech input 104 is likely to include explicit punctuation words. For example, when there is no default list of denormalizers or the default list does not include an explicit punctuation denormalizer, the denormalization instruction 410 can indicate that the denormalizer module 320 includes an explicit punctuation denormalizer. At time 5, the denormalizer module 320 denormalizes the raw speech recognition result 312 into a denormalized text 322 by sequentially applying the list of one or more denormalizers 350 to the generated raw speech recognition result 312.
[0049] At time 6, the ASR module 300 provides the denormalized text 322 corresponding to the speech input 104 to the user device 200, and the user device 200 displays the denormalized text 322 in the message / dictation window 218 displayed in the graphical user interface 214. In the example shown, the denormalized text 322 is denormalized by at least an explicit punctuation denormalizer and a capitalization denormalizer. For example, the explicit punctuation denormalizer converts the word "comma" in the original speech recognition result 312 into the corresponding symbol "," and recognizes the word "new line" as an instruction to start a new line from the beginning of "Thank you". After applying the explicit punctuation denormalizer, the denormalizer module 320 applies the capitalization denormalizer by capitalizing the first letter "D" of the first line, the first letter "C" of the recipient's name (carrie), and the first letter "T" of the second line. It is worth noting that if the capitalization denormalizer were applied before the explicit punctuation denormalizer, the "T" would most likely not be capitalized because it would not be identified as the first letter of a line.
[0050] Figure 2A and Figure 2B The example uses the exact same ASR module 300 with access to the same predefined set of denormalizers 350. The recognizer 310 is trained on the same normalized text and will generate the same raw speech recognition results 312 for a given speech input 104 regardless of what the associated contextual metadata 110 includes. However, the context server 400 uses the contextual metadata 110 associated with the speech input 104 to personalize / contextualize the denormalization of the raw speech recognition results 312 generated by the recognizer 310 into the denormalized text for optimal readability.
[0051] Figure 3An example interaction between the denormalizer module 320, the data storage 340, and the context server 400 of the ASR module 300 is illustrated. The data storage 340 stores a predefined set of denormalizers 350, each of which corresponds to a respective denormalization rule or denormalization model for denormalizing some aspect of the input of the normalized text received at the denormalizer module 320. When the context metadata 110 is not present and / or the context server 400 does not provide any denormalization instructions 410, the denormalizer module 320 may be initialized with a default list of denormalizers 348 to be applied in order to denormalize the speech recognition results 312. In the example shown, the default list of denormalizers 348 includes an explicit punctuation denormalizer, an implicit punctuation denormalizer, a capitalization denormalizer, and a literal denormalizer. Data store 340 maintains a key-value map 342 of a predefined set of denormalizers 350 that is accessible to denormalizer module 320 to select / retrieve any given denormalizer (ie, value 346 ) from data store 340 by providing a corresponding key 344 .
[0052] In response to receiving the denormalization instruction 410 from the context server 400, the denormalizer module 320 can select a list of one or more denormalizers 350 to be applied to denormalize the normalized text 322 by modifying the default list of denormalizers 348 based on the denormalization instruction 410. In some implementations, modifying the default list of denormalizers 348 includes removing one or more denormalizers from the default list of denormalizers 348 and / or adding one or more denormalizers from a predefined set of denormalizers 350 to the default list of denormalizers 348. For example, the denormalization instruction 410 includes one or more of the following: appending a denormalizer instruction, attaching a denormalizer instruction, removing a denormalizer instruction, replacing a denormalizer instruction, or adding a denormalizer instruction. Appending and attaching a denormalizer instruction can be beneficial when the context server 400 is not aware of the default list of denormalizers 348. The AppendDenormalizer instruction identifies a new denormalizer (e.g., from a predefined set of denormalizers 350) to be appended at the beginning of the default list of denormalizers 348, whereas the AppendDenormalizer identifies a new denormalizer to be appended at the end of the default list of denormalizers. The RemoveDenormalizer instruction identifies an existing denormalizer to be removed from the default list of denormalizers 348, and the ReplaceDenormalizer instruction identifies a new denormalizer to replace an existing denormalizer from the default list of denormalizers 348. The AddDenormalizer instruction identifies a new denormalizer to be added to the default list of denormalizers and identifies an existing denormalizer from the default list of denormalizers that precedes or follows the new denormalizer.
[0053] In some implementations, the context server 400 provides denormalization instructions 410 that include a remove denormalizer instruction that instructs the denormalizer module 320 to remove a transliteration denormalizer from a default list of the denormalizer 348 and an append denormalization instruction that instructs the denormalizer module 320 to add an offensive word denormalizer to the default list of the denormalizer 348. Here, the remove instruction includes a corresponding transliteration key 344 identifying the transliteration denormalizer to be removed from the default list of the denormalizer 348, and the append instruction includes a corresponding offensive word key 344 identifying the offensive word denormalizer to be added to the default list of the denormalizer 348.
[0054] In the example shown, the denormalizer module 320 is configured to modify the default list of denormalizers 348 using the denormalization instructions 410 received from the context server 400. For example, because the denormalization instructions 410 specify that the literal denormalizer should be removed and the offensive word denormalizer should be added, the denormalizer module 320 retrieves the remaining explicit punctuation denormalizers, implicit punctuation denormalizers, and capitalization denormalizers from the default list of denormalizers 348 and the offensive word denormalizer from the predefined set of denormalizers 350 from the key-value map 342 by providing the corresponding keys 344 to the data store 340.
[0055] In some examples, the denormalizer module 320 provides the corresponding keys 344 in an order associated with the order / sequence in which the corresponding denormalizers are to be applied in order to denormalize the raw speech recognition result 312 into the denormalized text 322. Thus, by providing the corresponding keys 344, the denormalizer module 320 retrieves a selected list of denormalizers 352 corresponding to the explicit punctuation denormalizer, the implicit punctuation denormalizer, the capitalization denormalizer, and the masking offensive denormalizer. Thereafter, the denormalizer module 320 denormalizes the raw speech recognition result 312 into the denormalized text 322 by applying the explicit punctuation denormalizer to the raw speech recognition result 312, applying the implicit punctuation denormalizer to the output of the explicit punctuation denormalizer, applying the capitalization denormalizer to the output of the implicit punctuation denormalizer, and applying the masking offensive denormalizer to the output of the capitalization denormalizer.
[0056] refer to Figure 4AIn some implementations, the context server 400 compiles a personalized denormalizer 450 for a specific user 10 that is not included in the predefined set of denormalizers 350 stored in the data storage device 340 (e.g., the memory hardware 206, 146) of the ASR module 300. Here, the user 10 or a third-party client can provide the context server 400 with one or more personalized denormalization rules 402 that the user 10 (or the third-party client) is willing to apply in order to denormalize the original speech recognition results 312. For example, the denormalization rules 402 can be implemented by the user 10 and include any personalized / custom denormalization rules specified by the user 10 for denormalizing text, such as special capitalization, rich fonts, embedding (e.g., links to websites), formatters that fit specific rendering constraints (e.g., shorter lines / tabs), etc.
[0057] In response to receiving one or more denormalization rules 402 (e.g., customized / personalized denormalization rules) from a specific user 10, the context server 400 compiles a corresponding personalized denormalizer 450 for each denormalization rule 402 and stores the one or more personalized denormalizers 450 in a personalized denormalizer repository 440, which may correspond to Figure 1 12 . The personalized denormalizer 450 specific to the user 10 may include a user identifier 12 that identifies the user 10. Thus, when the user 10 provides a subsequent ASR request 102 with context metadata 110 that includes the user identifier 12, the context server 400 may identify any personalized denormalizers 450 stored in the repository 440 (e.g., the context source 190) that match the user identifier 12. The context server 400 may then generate a denormalization instruction 410 that instructs the denormalizer module 320 to apply the personalized denormalizer 450. In some examples, the context server 400 encrypts the personalized denormalizer 450 associated with the user 10 and applies a cryptographic hash 14 that identifies a cryptographic key that the user 10 must provide in order for the denormalizer module 320 to decrypt and use the personalized denormalizer 450. For example, the user 10 may provide the cryptographic key as part of the context metadata 110 in the ASR request 102 for converting the corresponding speech input 104 into the denormalized text 322 .
[0058] In some implementations, the personalized denormalizer repository 440 is separate from the ASR module 300, enabling personalized denormalizers 450 (e.g., custom denormalizers) to be dynamically compiled and stored without having to wait for a periodic update to the ASR module 300. Additionally, because the personalized denormalizer repository 440 may potentially store one or more respective sets of personalized denormalizers 450 for millions of different users, it is not ideal for the data store 340 at the ASR module 300 to also store and provide access to all of these personalized denormalizers 450 in addition to the predefined set of denormalizers 350 that are accessible to the entire user population. In some scenarios, the personalized denormalizer repository 440 may store the personalized denormalizers 450 and push the personalized denormalizers to the data store 340 of the ASR module 300 during the next periodic (e.g., weekly) update of the ASR module 300. In some examples, personalized denormalizer repository 440 is integrated with data store 340 on ASR module 300 , without mixing personalized denormalizer 450 with a predefined set of denormalizers 350 .
[0059] refer to Figure 4B In some implementations, the user 10 provides an ASR request 102 via the user device 200, the ASR request 102 including one or more personalized denormalization rules 402 as part of the context metadata 110 of the ASR request 102. In this scenario, the user 10 is requesting the ASR module 300 to apply the one or more denormalization rules 402 on the fly. In the example shown, the context metadata 110 including the one or more personalized denormalization rules 402 is provided to the context server 400 and the speech input 104 is provided to the recognizer 310. The recognizer 310 generates a raw speech recognition result 312 corresponding to the speech input 104. At the same time, the context server 400 is configured to compile a corresponding personalized denormalizer 450 for each denormalization rule 402 and generate a denormalization instruction 410 including the personalized denormalizer 450 for denormalizing the raw speech recognition result 312 corresponding to the speech input 104. Therefore, the context server 400 provides the denormalization instruction 410 including the personalized denormalizer 450 to the denormalizer module 320 , and the denormalizer module 320 applies the personalized denormalizer 450 on the fly to denormalize the original speech recognition result 312 into the corresponding denormalized text 322 .
[0060] In some implementations, the user 10 may include an administrator of the ASR module 300 who provides personalized denormalization rules 402 associated with a new denormalizer that the administrator wants to test without having to update a predefined set of denormalizers 350 or other aspects of the ASR module 300. For example, the personalized denormalization rules 402 may cause the context server 400 to compile the new denormalizer and generate instructions 410 that instruct the denormalizer module 320 to use the new denormalizer for denormalizing text. Thus, providing personalized denormalization rules 402 on the fly is valuable for debugging new denormalizers before they are pushed out to the ASR module 300.
[0061] Figure 6 6 is a flow chart of an example arrangement of operations of a method 600 for denormalizing raw speech recognition results generated by a denormalizer module 320 of an ASR module 300. At operation 602, the method 600 includes receiving speech input 104 from a user 10 at data processing hardware 204, 144 of the ASR module 300. The user 10 may be associated with a user device 200 that provides the speech input 104 to the ASR module 300 as part of a speech recognition request 102. The ASR module 300 may be located on the user device 200 and / or the distributed system 140.
[0062] At operation 604, method 600 includes obtaining, by data processing hardware 204, 144, contextual metadata 110 associated with speech input 104. Contextual metadata 110 may be received from user device 200 when audio data 103 of speech input 104 is received at data processing hardware 204, 144. In other examples, obtaining contextual metadata 110 includes: receiving a user identifier 12 associated with user 10 when speech input 104 is received; and retrieving contextual metadata 110 from memory hardware 206, 146 using user identifier 12. Contextual metadata may include at least one of: a speech recognition category associated with speech input (e.g., dictation or voice search), an application running on user device 200, a location of the user when speech input 104 is received, user identifier 12, demographic information associated with the user, or user preferences for denormalizing text. For example, user preferences may include one or more personalized denormalization rules 402.
[0063] At operation 606, the method 600 includes generating, by the data processing hardware 204, 144, a raw speech recognition result 312 corresponding to the speech input 104. The generated raw speech recognition result 312 includes normalized text. For example, normalized text refers to text from which all capitalization and punctuation are stripped. For example, because the recognizer 310 of the ASR module 300 is typically trained on a corpus of normalized text, the recognizer 310 will output the normalized raw speech recognition result 312.
[0064] At operation 608, the method 600 includes selecting a list of one or more denormalizers 352 to be applied to the generated raw speech recognition result 312 result based on the context metadata 110 associated with the speech input 104. Selecting the list of one or more denormalizers 352 may include modifying a default list of denormalizers 348 from a predefined set of denormalizers 350 by at least one of removing one or more denormalizers from the default list of denormalizers 348 or adding one or more denormalizers from the predefined set of denormalizers 350 to the default list of denormalizers.
[0065] At operation 610, the method 600 includes denormalizing the generated raw speech recognition result 312 into a denormalized text 322 by sequentially applying the list of one or more denormalizers 352 to the generated raw speech recognition result 312. In some examples, the context server 400 generates a denormalization instruction 410 based on the context metadata 110 for denormalizing the generated raw speech recognition result 312 into a denormalized text 322. Here, the denormalizer module 320 of the ASR module 300 can receive the denormalization instruction 410 from the context server 400 and use the denormalization instruction 410 to select the list of one or more denormalizers 352. For example, the denormalizer module 320 can use the corresponding key 344 to retrieve each denormalizer in the selected list of one or more denormalizers 350 from the memory hardware (e.g., the data storage device 340). The denormalization instruction 410 can identify the desired denormalizer to be selected by providing its corresponding key 344. Thereafter, the denormalizer module 320 is configured to sequentially apply each denormalizer in the selected list of one or more denormalizers 352 retrieved from the memory hardware 340 to the generated raw speech recognition result 312 .
[0066] The ASR module 300 may transmit the denormalized text 322 to an output 500. The output 500 may include the above reference Figure 5Any of the outputs 500 described. For example, the output 500 may include a client device such as the user device 200 or another user device displaying the denormalized text 322 on the display screen 212 of the client device 200. In some examples, the denormalized text 322 is streamed to the output 500 in real time.
[0067] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.
[0068] Non-transitory memory may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-volatile memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as root flash programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.
[0069] Figure 7 700 that can be used to implement the systems and methods described in this document. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.
[0070] The computing device 700 includes a processor 710 (data processing hardware), a memory 720 (memory hardware), a storage device 730, a high-speed interface / controller 740 connected to the memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and the storage device 730. Each of the components 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be mounted on a common motherboard or otherwise as appropriate. The processor 710 (e.g., data processing hardware 204, 144) is capable of processing instructions for execution within the computing device 700, including instructions stored in the memory 720 (e.g., memory hardware 206, 146) or on the storage device 730 (e.g., memory hardware 206, 146) to be displayed on a display device such as a display 780 (e.g., coupled to the high-speed interface 740). Figure 2A and Figure 2B The processor 710 may be configured to execute instructions for displaying graphical information of a graphical user interface (GUI) on an external input / output device (e.g., screen 212). In some implementations, the processor 710 executes the ASR module 300 and the context server 400. In other implementations, multiple processors and / or multiple buses and multiple memories and memory types may be used as appropriate. In addition, multiple computing devices 700 may be connected, each of which provides a portion of the necessary operations (e.g., as a server group, a group of blade servers, or a multi-processor system).
[0071] The memory 720 stores information non-temporarily within the computing device 700. The memory 720 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 720 may be a physical device for storing programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as root flash programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.
[0072] The storage device 730 can provide mass storage for the computing device 700. In some implementations, the storage device 730 is a computer readable medium. In various implementations, the storage device 730 can be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state storage device, or a device array, including devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as the above-mentioned methods. The information carrier is a computer or machine readable medium, such as a memory 720, a storage device 730, or a memory on a processor 710.
[0073] The high-speed controller 740 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 760 manages less bandwidth-intensive operations. This allocation of responsibilities is exemplary only. In some implementations, the high-speed controller 740 is coupled to the memory 720, to the display 780 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 750, which can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to the storage device 730 and the low-speed expansion port 790. The low-speed expansion port 790, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or to a networking device such as a switch or a router, for example, through a network adapter.
[0074] As shown, computing device 700 can be implemented in many different forms. For example, it can be implemented as a standard server 700a or multiple times in a group of such servers 700a, as a laptop computer 700b or as a part of a rack server system 700c. Various implementations of the systems and techniques described herein can be implemented with digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which can be dedicated or general, coupled to receive data and instructions from a storage system, at least one input device and at least one output device, and transmit data and instructions to the storage system, at least one input device and at least one output device.
[0075] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0076] The processes and logic flows described in this specification can be performed by executing one or more computer programs through one or more programmable processors (also referred to as data processing hardware) to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by special logic circuits such as FPGA (field programmable gate array) or ASIC (application-specific integrated circuit). As an example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The necessary elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be coupled in operation to receive data from the mass storage device or transfer data to the mass storage device, or both. However, a computer does not have to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including by way of example: semiconductor memory devices (e.g., EPROM, EEPROM) and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuits.
[0077] To provide interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, and optionally having a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.
[0078] Many implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.
Claims
1. A method, include: At data processing hardware of a user device, an automatic speech recognition (ASR) request is received from a user of the user device, the ASR request comprising: voice input from the user; and contextual metadata indicating that the speech input is associated with a voice search application currently executing on the user device; generating, by the data processing hardware, an original speech recognition result corresponding to the speech input using a speech recognizer; normalizing, by the data processing hardware, the generated raw speech recognition results into text suitable for processing by a natural language understanding (NLU) module based on the contextual metadata indicating that the speech input is associated with the voice search application currently executing on the user device; and The text is processed by the data processing hardware using the NLU module to understand the speech input.
2. The method according to claim 1, in, The voice input is captured by a microphone of the user device.
3. The method according to claim 1, in, The context metadata includes an application identifier indicating that the voice search application is currently executing on the user device.
4. The method according to claim 1, in, The contextual metadata includes a speech recognition category indicating that the speech input is associated with a voice command to be executed by the user device.
5. The method according to claim 1, further comprising: include: A list of one or more denormalizers to be applied to the generated raw speech recognition results is selected by the data processing hardware based on the context metadata.
6. The method according to claim 5, in, Selecting the list of the one or more denormalizers includes modifying the default list of denormalizers from a predefined set of denormalizers by removing one or more denormalizers from the default list of denormalizers.
7. The method according to claim 5, in, Selecting the list of the one or more denormalizers includes modifying a default list of denormalizers from the predefined set of denormalizers by adding one or more denormalizers from the predefined set of denormalizers to the default list of denormalizers.
8. The method according to claim 5, in, Selecting the list of one or more denormalizers comprises: retrieving each denormalizer in the selected list of the one or more denormalizers from storage hardware in communication with the data processing hardware using a corresponding key; and Each denormalizer in the selected list of one or more denormalizers retrieved from the memory hardware is sequentially applied to the generated raw speech recognition results.
9. The method according to claim 8, in, The memory hardware is located on the user equipment.
10. The method according to claim 1, in, The speech recognizer and the NLU module are executed on the data processing hardware of the user device.
11. A user device, include: Data processing hardware; as well as Memory hardware that communicates with the data processing hardware and stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including: An automatic speech recognition (ASR) request is received from a user of the user device, the ASR request comprising: voice input from the user; and contextual metadata indicating that the speech input is associated with a voice search application currently executing on the user device; Using a speech recognizer to generate an original speech recognition result corresponding to the speech input; normalizing the generated raw speech recognition results into text suitable for processing by a natural language understanding (NLU) module based on the context metadata indicating that the speech input is associated with the voice search application currently executing on the user device; and The text is processed using the NLU module to understand the speech input.
12. The user equipment according to claim 11, in, The voice input is captured by a microphone of the user device.
13. The user equipment according to claim 11, in, The context metadata includes an application identifier indicating that the voice search application is currently executing on the user device.
14. The user equipment according to claim 11, in, The contextual metadata includes a speech recognition category indicating that the speech input is associated with a voice command to be executed by the user device.
15. The user equipment according to claim 11, in, The operations further include selecting a list of one or more denormalizers to be applied to the generated raw speech recognition results based on the context metadata.
16. The user equipment according to claim 15, in, Selecting the list of the one or more denormalizers includes modifying the default list of denormalizers from a predefined set of denormalizers by removing one or more denormalizers from the default list of denormalizers.
17. The user equipment according to claim 15, in, Selecting the list of the one or more denormalizers includes modifying a default list of denormalizers from the predefined set of denormalizers by adding one or more denormalizers from the predefined set of denormalizers to the default list of denormalizers.
18. The user equipment according to claim 15, in, Selecting the list of one or more denormalizers comprises: retrieving each denormalizer in the selected list of the one or more denormalizers from storage hardware in communication with the data processing hardware using a corresponding key; and Each denormalizer in the selected list of one or more denormalizers retrieved from the memory hardware is sequentially applied to the generated raw speech recognition results.
19. The user equipment according to claim 11, in, The speech recognizer executes on the data processing hardware of the user device.
20. The user equipment according to claim 11, in, The NLU module is executed on the data processing hardware of the user device.
21. A computer-implemented method which, when executed on data processing hardware, causes the data processing hardware to perform an operation, the operation include: receiving voice input from a user; obtaining contextual metadata associated with the speech input, the contextual metadata indicating that the speech input includes spoken speech directed to a messaging application currently executing on a user device for inclusion in an electronic message; generating, using a speech recognizer, an original speech recognition result corresponding to the speech input, the original speech recognition result including explicit punctuation words spoken by the user; denormalizing the generated raw speech recognition results into denormalized text by applying an explicit punctuation denormalizer to convert explicitly punctuated words in the raw speech recognition results into corresponding punctuation marks based on the contextual metadata indicating that the speech input includes spoken speech directed to the messaging application; as well as The denormalized text including the corresponding punctuation marks is displayed on a display screen of the user device.
22. The method according to claim 21, in, The voice input is captured by a microphone of the user device.
23. The method according to claim 22, in, The user device includes a desktop computing device, a laptop, a tablet, a smart phone, or a wearable computing device.
24. The method according to claim 21, in, The data processing hardware is located on the user equipment.
25. The method according to claim 21, in, The operations further include selecting a list of one or more denormalizers to be applied to the generated raw speech recognition results based on the context metadata.
26. The method according to claim 25, in, Selecting the list of the one or more denormalizers includes modifying the default list of denormalizers by removing one or more denormalizers from the default list of denormalizers.
27. The method according to claim 25, in, Selecting the list of the one or more denormalizers includes modifying the default list of denormalizers by adding one or more denormalizers from a predefined set of denormalizers to the default list of denormalizers.
28. The method according to claim 25, in, Selecting the list of one or more denormalizers comprises: retrieving each denormalizer in the selected list of the one or more denormalizers from storage hardware in communication with the data processing hardware using a corresponding key; and Each denormalizer in the selected list of one or more denormalizers retrieved from the memory hardware is sequentially applied to the generated raw speech recognition results.
29. The method according to claim 28, in, The memory hardware is located on the user equipment.
30. The method according to claim 21, in, The speech recognizer executes on the data processing hardware.
31. The method according to claim 21, in, Denormalizing the generated raw speech recognition results into the denormalized text is further based on a user preference for denormalizing the text.
32. A user device, include: Display screen; data processing hardware in communication with the display screen; as well as Memory hardware that communicates with the data processing hardware and stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including: receiving voice input from a user; obtaining contextual metadata associated with the speech input, the contextual metadata indicating that the speech input includes spoken speech directed to a messaging application currently executing on a user device for inclusion in an electronic message; generating, using a speech recognizer, an original speech recognition result corresponding to the speech input, the original speech recognition result including explicit punctuation words spoken by the user; Based on the context metadata indicating that the speech input includes spoken speech directed to the messaging application, denormalizing the generated raw speech recognition results into denormalized text by applying an explicit punctuation denormalizer to convert explicitly punctuated words in the raw speech recognition results into corresponding punctuation marks; and The denormalized text including the corresponding punctuation marks is displayed on the display screen.
33. The user equipment according to claim 32, in, The voice input is captured by a microphone of the user device.
34. The user equipment according to claim 32, in, The user device includes a desktop computing device, a laptop, a tablet, a smart phone, or a wearable computing device.
35. The user equipment according to claim 32, in, The operations further include selecting a list of one or more denormalizers to be applied to the generated raw speech recognition results based on the context metadata.
36. The user equipment according to claim 35, in, Selecting the list of the one or more denormalizers includes modifying the default list of denormalizers by removing one or more denormalizers from the default list of denormalizers.
37. The user equipment according to claim 35, in, Selecting the list of the one or more denormalizers includes modifying the default list of denormalizers by adding one or more denormalizers from a predefined set of denormalizers to the default list of denormalizers.
38. The user equipment according to claim 35, in, Selecting the list of one or more denormalizers comprises: retrieving each denormalizer in the selected list of the one or more denormalizers from storage hardware in communication with the data processing hardware using a corresponding key; and Each denormalizer in the selected list of one or more denormalizers retrieved from the memory hardware is sequentially applied to the generated raw speech recognition results.
39. The user equipment according to claim 32, in, The speech recognizer executes on the data processing hardware.
40. The user equipment according to claim 32, in, Denormalizing the generated raw speech recognition results into the denormalized text is further based on a user preference for denormalizing the text.
