Detecting and restoring out-of-vocabulary words in speech-to-text transcription system

Abstract

There is provided a method for restoring out-of-vocabulary words in a transcription of a voice data record, comprising: receiving a voice data record for transcription as a textual representation of the voice data record; transcribing the voice data record into the text representation using a word recognition model; identifying unknown words in the text representation; reconstructing the unknown word in the textual representation based on identification of subunits of the unknown word generated by a subunit identification model; modifying the textual representation of the speech data record by replacing the unknown word with the reconstruction of the unknown word; and outputting the modified textual representation of the voice data record.

Description

[0001] This application is a divisional application of the application for invention patent application number 202080062694.5, filed on September 10, 2020, entitled “Detecting and Recovering Out-of-Vocabulary Words in a Voice-to-Text Transcription System”. TECHNICAL FIELD

[0002] The present invention relates to voice-to-text transcription systems, and more specifically, to identifying and recovering out-of-vocabulary words in voice data using a text recognition model. BACKGROUND

[0003] Voice-to-text systems are used in various settings to convert audio content into textual content. For example, a voice-to-text system can be used to generate a transcription of spoken words in real-time for future reference or official record (e.g., transcription of a legal proceeding, a meeting, etc.). In another example, a voice-to-text system can be used in real-time to augment audio or audiovisual content with a textual representation of the audio, such as a live broadcasted television program.

[0004] Generally, a voice-to-text system can be trained to generate a textual representation of audio content by mapping portions of the audio content to whole words. Depending on the size and content of the dictionary used to train the voice-to-text system, the voice-to-text system can have difficulty accurately constructing a transcription of the audio content. For example, if the voice-to-text system is trained on a small dictionary, a large number of words can not be accurately recognized. In another example, specialized, domain-specific words (e.g., scientific jargon) can not be accurately recognized by a voice-to-text system trained using a general knowledge dictionary.

[0005] When a voice-to-text system encounters an out-of-vocabulary word, or a word that is not found in the dictionary used to train the voice-to-text system, the voice-to-text system can output a null string for the unknown word or attempt to output a known word that sounds like a portion of the unknown word. Outputting a null string can result in a loss of information between the audio content and the textual representation of the audio content. Meanwhile, outputting a known word that sounds like a portion of the unknown word can result in generating a nonsensical textual representation of the audio content.

[0006] Accordingly, there is a need in the art to address the above-referenced problems. SUMMARY

[0007] From a first aspect, the present invention provides a method for recovering out-of-vocabulary words in a transcription of a voice data recording, comprising: receiving a voice data recording for transcription into a textual representation of the voice data recording; transcribing the voice data recording into the textual representation using a word recognition model; identifying an unknown word in the textual representation; reconstructing the unknown word in the textual representation based on recognition of sub-units of the unknown word generated by a sub-unit recognition model; modifying the textual representation of the voice data recording by replacing the unknown word with the reconstruction of the unknown word; and outputting the modified textual representation of the voice data recording.

[0008] From another aspect, the present invention provides a system, comprising: a processor; and a memory having stored thereon instructions that, when executed by the processor, perform operations for recovering out-of-vocabulary words in a transcription of a voice data recording, the operations comprising: receiving a voice data recording for transcription into a textual representation of the voice data recording; transcribing the voice data recording into the textual representation using a word recognition model; identifying an unknown word in the textual representation; reconstructing the unknown word in the textual representation based on recognition of sub-units of the unknown word generated by a sub-unit recognition model; modifying the textual representation of the voice data recording by replacing the unknown word with the reconstruction of the unknown word; and outputting the modified textual representation of the voice data recording.

[0009] From another aspect, the present invention provides a computer program product for recovering out-of-vocabulary words in a transcription of a voice data recording, the computer program product comprising a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method for performing the steps of the present invention.

[0010] From another aspect, the present invention provides a computer program stored on a computer readable medium and loadable into the internal memory of a digital computer, comprising software code portions, when said program is run on a computer, for performing the steps of the present invention.

[0011] From another aspect, the present invention provides a computer readable medium having stored thereon instructions, which, when executed by a processor, perform operations for recovering out-of-vocabulary words in a transcription of a voice data recording, the operations comprising: receiving a voice data recording for transcription into a textual representation of the voice data recording; transcribing the voice data recording into the textual representation using a word recognition model; identifying an unknown word in the textual representation; reconstructing the unknown word in the textual representation based on recognition of sub-units of the unknown word generated by a sub-unit recognition model; modifying the textual representation of the voice data recording by replacing the unknown word with the reconstruction of the unknown word; and outputting the modified textual representation of the voice data recording.

[0012] One aspect of the disclosure provides a method for recovering out-of-vocabulary words in a transcription of speech data. The method generally includes receiving a speech data recording for transcription into a textual representation of the speech data recording. The speech data recording is transcribed into the textual representation using a word recognition model. An unknown word is identified in the textual representation, and the unknown word is reconstructed based on recognition of sub-units of the unknown word generated by a sub-unit recognition model. The textual representation of the speech data recording is modified by replacing the unknown word with the reconstruction of the unknown word, and the modified textual representation is output.

[0013] Another aspect of the disclosure provides a system having a processor and a memory. The memory generally has instructions stored thereon that, when executed by the processor, perform operations for recovering out-of-vocabulary words in a transcription of speech data. The operations generally include receiving a speech data recording for transcription into a textual representation of the speech data recording. The speech data recording is transcribed into the textual representation using a word recognition model. An unknown word is identified in the textual representation, and the unknown word is reconstructed based on recognition of sub-units of the unknown word generated by a sub-unit recognition model. The textual representation of the speech data recording is modified by replacing the unknown word with the reconstruction of the unknown word, and the modified textual representation is output.

[0014] Yet another aspect of the disclosure provides a computer-readable medium having instructions stored thereon that, when executed by a processor, perform operations for recovering out-of-vocabulary words in a transcription of speech data. The operations generally include receiving a speech data recording for transcription into a textual representation of the speech data recording. The speech data recording is transcribed into the textual representation using a word recognition model. An unknown word is identified in the textual representation, and the unknown word is reconstructed based on recognition of sub-units of the unknown word generated by a sub-unit recognition model. The textual representation of the speech data recording is modified by replacing the unknown word with the reconstruction of the unknown word, and the modified textual representation is output. BRIEF DESCRIPTION OF DRAWINGS

[0015] The application will now be described by way of example only with reference to preferred embodiments, as illustrated in the following drawings:

[0016] Figure 1 A networked computing environment in which audio content is transcribed into a textual representation using a first text recognition model and unknown words in the textual representation are recovered using a second text recognition model is shown in accordance with an embodiment.

[0017] Figure 2 An example operation for generating a textual representation of audio content using a first text recognition model and recovering out-of-vocabulary words in the generated textual representation using a second text recognition model is shown in accordance with an embodiment.

[0018] Figure 3Example operations for recovering out-of-vocabulary words using a connectionist temporal classification (CTC) text recognition model are shown in accordance with an embodiment.

[0019] Figure 4 Example operations for recovering out-of-vocabulary words using a sequence-to-sequence text recognition model are shown in accordance with an embodiment.

[0020] Figure 5 An example system in which aspects of the disclosure can be implemented is shown. DETAILED DESCRIPTION

[0021] Out-of-vocabulary words in audio content generally have an adverse effect on the accuracy and readability of transcripts produced by audio-to-text transcription systems. Because out-of-vocabulary words have a negative impact on transcripts, various techniques have been developed to address the presence of out-of-vocabulary words in audio content. In one example, a hybrid speech recognition system can use various decoding outputs that can compensate for the presence of out-of-vocabulary words and attempt to improve the accuracy and readability of transcripts. These outputs can include word confidence information and timing information that can be used to identify the location and length of out-of-vocabulary words and attempt to recover these words. For example, a speech recognition model can use sub-word units to explicitly model out-of-vocabulary words or can use confidence scores produced by the speech recognition system to classify text output into out-of-vocabulary and in-vocabulary regions. In other examples, an end-to-end neural network-based speech recognition system can be used to identify and attempt to recover out-of-vocabulary words, which uses a training and inference pipeline that can be simpler than those used in conventional speech recognition systems.

[0022] However, these systems have limitations that reduce the performance of these systems in identifying and recovering out-of-vocabulary words in audio content. Systems that use primarily whole-word based recognition systems and character-based models for out-of-vocabulary recognition and recovery can rely on confidence scores generated by the whole-word based recognition system, which can result in the generation of a reserved unknown word token that ultimately invokes the character-based model for a word that can actually be known by the whole-word based recognition system. Further, these systems can produce inaccurate timing information for words in audio content or produce no timing information, which can reduce the performance of out-of-vocabulary word detection in the speech recognition system.

[0023] Aspects of the present disclosure provide efficient techniques for detecting and recovering out-of-vocabulary words in audio content. As discussed in further detail below, an out-of-vocabulary word can be detected by determining whether a speech recognition system has reconstructed an entire word or a collection of word fragments (e.g., phonetic segments, characters, etc.). If the speech recognition system has reconstructed the word as a collection of word fragments, the speech recognition system can consider the word to be an unknown word and attempt to recover the word from the collection of word fragments. Through the use of the determination of whether a word has been reconstructed as an entire word or a collection of word fragments, aspects discussed herein can bypass the use of confidence information to determine whether a word is in- or out-of-vocabulary. The speech recognition system can additionally or alternatively use an entire word sequence-to-sequence system, such as an attention-based model, to identify in- and out-of-vocabulary words and reconstruct an out-of-vocabulary word using character output in a region overlapping the out-of-vocabulary word.

[0024] Figure 1 An example networked computing environment in which an entire word and word fragment recognition model is used to identify and recover out-of-vocabulary words in a transcription of audio content is shown in accordance with an embodiment of the present disclosure. As shown, the computing environment 100 includes a client device 120, a speech recognition system 130, and a speech-to-text data store 140.

[0025] The client device 120 generally represents a computing device on which a user can access audiovisual content and receive a textual representation of the audiovisual content from the speech recognition system 130 for display. As shown, the client device 120 includes a software application 122.

[0026] The software application 122 generally provides an interface through which a user can access audiovisual content from a remote source and invoke speech recognition and transcript generation operations at the speech recognition system 130. In some embodiments, the software application 122 can provide a user of the client device 120 with a mechanism to upload previously recorded audiovisual files to the speech recognition system 130 and retrieve transcripts of recorded audiovisual files therefrom. In some embodiments, the software application 122 can be a web browser or other application that allows a user to interact with the speech recognition system 130.

[0027] The speech recognition service 130 receives audiovisual content and generates a transcript of the audiovisual content using a word and word subunit (or word piece) recognition model and outputs the transcript to a client application for display and / or to a data store (e.g., the speech-to-text data store 140) for future retrieval. As shown, the speech recognition system 130 includes a speech data transcriber 132 and an out-of-vocabulary word recoverer 134.

[0028] The speech data transcriber 132 generally processes received audiovisual content through a word recognition model and a sub-word unit recognition model to generate a textual representation of the audio content, such as a speech data recording or an audiovisual file that includes a speech data track. The models used by the speech data transcriber 132 can be trained by a remote system (not shown) and deployed to the speech recognition system 130 for use in generating transcripts of speech data recordings, and these models can be periodically updated to improve the accuracy of transcripts generated using these models. In some embodiments, a word recognition model can be trained using a training dataset of entire words and word sub-units, such as phonemes or portions of entire words. As discussed in further detail below, where a word recognition model is trained using a training dataset of entire words and word sub-units, a plurality of word sub-units that generate a given portion of a speech data recording can be used as an indication that the given portion of the speech data recording corresponds to an out-of-vocabulary word. A word sub-unit recognition model can be trained using a training dataset of word portions, such as phonemes, portions of words (e.g., syllables), characters, or other components of words that can be included in a speech data recording. As discussed in further detail below, the output of a word sub-unit recognition model can be used to recover an out-of-vocabulary word.

[0029] The word recognition model and the word sub-unit recognition model can be implemented as neural networks configured to map an audio representation of a word (or a portion of a word) to a corresponding textual representation of the word (or the portion of the word). In some embodiments, where the word recognition model and the word sub-unit recognition model are connectionist temporal classification models, the word recognition model can be configured to map a portion of a speech data recording that records a word or a word sub-unit, and the sub-unit recognition model can be configured to map the portion of the speech data recording to one or more phoneme or syllable word sub-units that can be concatenated with other word sub-units to recover an out-of-vocabulary word. In some embodiments, where the word recognition model and the word sub-unit recognition model are sequence-to-sequence models, the word recognition model can be configured to output timing information and map a portion of a speech data recording to an entire word or an indication that the word is unknown to the word recognition model to preserve the sequence, and the word sub-unit recognition model can be configured to map the portion of the speech data recording to one or more characters and output timing information for each character recognized in the speech data recording.

[0030] The speech data transcriber 132 generally receives a speech data recording as an audio file or as an audio track in an audiovisual file and transcribes the speech data recording into a textual representation of the speech data recording. To transcribe the speech data recording into the textual representation, the speech data transcriber 132 can input the speech data recording into a word recognition model that generates a textual output for a given audio input. As discussed above, the word recognition model can be a neural network that is configured to map information in the speech data recording to individual whole words or portions of words, e.g., based on a confidence associated with a mapping of a portion of the speech data recording to a word or portion of a word in an overall set of words and word portions used to train the word recognition model, that has the highest probability in a probability distribution over the overall set of words and word portions used to train the word recognition model. For example, the word recognition model can generate a probability distribution over the overall set of words and word portions used to train the word recognition model and select the word or portion of a word in the probability distribution that has the highest probability as the word or portion of a word that corresponds to each portion of the speech data recording.

[0031] For a word recognition model that is structured as a connectionist temporal classification model, the word recognition model can cause a portion of the speech data recording to be mapped to a whole word or a word fragment, such as a phonetic segment, a syllable component of a word, etc. Generally, mapping a portion of the speech data recording to a whole word can indicate that the word recognition model identified the portion of the speech data recording as a word within the vocabulary, while mapping the portion of the speech data recording to a word fragment can indicate that the word recognition model identified the portion of the speech data recording (and other adjacent portions of the speech data recording) as a word outside the vocabulary. To determine whether a portion of the speech data recording has been mapped to a whole word or a word portion, the speech data transcriber 132 can search a dictionary for an exact match to each word or word portion that is mapped to the portion of the speech data recording. Groupings of consecutive word portions that can not exist in the dictionary can thus correspond to words outside the vocabulary for reconstruction by the words outside the vocabulary restorer 134, as discussed below.

[0032] For a word recognition model that is structured as a sequence-to-sequence model, the word recognition model can generate a word within the vocabulary for a portion of the speech data recording that is mapped to a recognized word in a dictionary used to train the sequence-to-sequence model. Conversely, a reserved character sequence or a reserved word subunit can be generated for a portion of the speech data recording that is not mapped to a recognized word in the dictionary. As discussed in further detail below, the reserved character sequence or the reserved word subunit, such as “ <unk>The out-of-vocabulary word restorer 134 can be configured to receive the transcript of the speech data recording from the speech data transcriber 132 and to identify the out-of-vocabulary words in the transcript. The out-of-vocabulary word restorer 134 can be configured to use a word sub-unit recognition model to identify the out-of-vocabulary words in the transcript. The out-of-vocabulary word restorer 134 can be configured to use a sequence-to-sequence model to reconstruct the out-of-vocabulary words from the sequence of word sub-units identified by the word sub-unit recognition model. The out-of-vocabulary word restorer 134 can be configured to use a connectionist temporal classification model to identify the out-of-vocabulary words in the transcript.

[0033] The out-of-vocabulary word restorer 134 generally receives the transcript of the speech data recording from the speech data transcriber 132 and uses a corresponding word sub-unit recognition model to reconstruct the out-of-vocabulary words identified by the word recognition model. The techniques used by the out-of-vocabulary word restorer 134 can differ based on whether the word recognition model and the word sub-unit recognition model are connectionist temporal classification models, sequence-to-sequence models, or other language recognition models that can be used to transcribe the speech data recording into a textual representation of the speech data recording.

[0034] For a word recognition model structured as a connectionist temporal classification model, each out-of-vocabulary word identified in the transcript of the speech data recording can be defined as a sequence of contiguous word sub-units or portions bounded by a preceding recognized whole word and a succeeding recognized whole word. To restore the out-of-vocabulary words, the out-of-vocabulary word restorer 134 is generally configured to use a word sub-unit recognition model that is also structured as a connectionist temporal classification model and is trained to map portions of the speech data recording to word sub-units, such as speech segments, word syllables, etc. The out-of-vocabulary word restorer 134 can identify the word sub-unit segments generated by the word sub-unit recognition model that correspond to the out-of-vocabulary words in the transcript and join the word sub-units together into one or more word sub-unit groups and reconstruct the words from the one or more word sub-unit groups. In some embodiments in which the word sub-units are speech segments, the out-of-vocabulary word restorer 134 can search a reverse speech dictionary to reconstruct the words from each grouping of speech segments. In other embodiments in which the word sub-units are syllables or other non-speech word sub-units, the out-of-vocabulary word restorer 134 can join the word sub-units in the group into a single string and search a dictionary for potential matches to the string. The out-of-vocabulary word restorer 134 can use fuzzy matching techniques, such as those used in computer-based spelling correction mechanisms, to identify likely matches to the string as words that correspond to the word sub-unit group. After identifying the words from the word sub-unit group, the out-of-vocabulary word restorer 134 can replace the word sub-unit group in the transcript generated by the speech data transcriber 132 with the identified words, thereby restoring the out-of-vocabulary words in the speech data recording with accurate representations of the out-of-vocabulary words.

[0035] In embodiments where the word recognition model is a sequence-to-sequence model, out-of-vocabulary words and special character sequences that identify out-of-vocabulary words as unknown words in the transcript generated by the speech data transcriber 132 can be accompanied by an attention peak. The attention peak can be a timestamp in the speech data recording around which the unknown, out-of-vocabulary word is located. The out-of-vocabulary word can be recovered on a per-character basis using a word sub-unit recognition model, which can be a sequence-to-sequence model configured to recognize individual characters in the speech data recording. To identify the characters that correspond to the out-of-vocabulary word, the out-of-vocabulary word recoverer 134 can use the attention peak associated with the out-of-vocabulary word to identify the likely boundaries of the word in the speech data recording that is provided to the speech recognition system 130 for analysis. The boundaries of the word can be defined based on the identification of whitespace characters within the speech data recording, where, in the speech data recording, the first character of the word is identified immediately after a whitespace character and the last character of the word is identified immediately before the next whitespace character. In some embodiments, surrounding context can also or instead be used to identify the characters associated with the out-of-vocabulary word in the speech data recording. For example, information about the characters in the words that precede and follow the out-of-vocabulary word can be used to identify the boundaries of the out-of-vocabulary word in the per-character transcript of the speech data recording generated by the word sub-unit recognition model.

[0036] In general, each character identified in the speech data recorded by the word sub-unit recognition model can be accompanied by a corresponding attention peak in the model. The out-of-vocabulary word recoverer 134 can generate a string by concatenating the characters associated with each attention peak in the model between the whitespace characters or other boundary characters identified for the out-of-vocabulary word. The string generated by the out-of-vocabulary word recoverer 134 can replace the sequence of reserved characters for the unknown word in the transcript of the speech data recording.

[0037] After the out-of-vocabulary word recoverer 134 modifies the transcript of the speech data recording by replacing the sequence associated with the out-of-vocabulary word with a reconstruction of the out-of-vocabulary word using the word sub-unit recognition model, the out-of-vocabulary word recoverer 134 outputs the modified transcript. In some embodiments, the modified transcript can be output for display in the software application 122 executing on the client device 120 in real-time or near real-time with the audiovisual content from which the transcript was generated. In some embodiments, the modified transcript can be submitted to the speech-to-text data store 140 for future reference.

[0038] In some embodiments, where the speech recognition system 130 uses connectionist temporal classification word recognition and word subunit recognition models, the interpolation weights associated with these models can be adjusted to control the likelihood that a word will be recognized as an out-of-vocabulary word. Higher interpolation weights associated with the word subunit recognition models can increase the number of words that are recognized as out-of-vocabulary words and reconstructed from word subunits (e.g., phonetic units, syllable units, etc.), which can reduce the accuracy of the system while increasing the likelihood that the word can be recalled using the word subunit analysis and reconstruction techniques discussed above. Similarly, lower interpolation weights associated with the word subunit recognition model can reduce the number of words that are recognized as out-of-vocabulary words, which can reduce the number of words that are reconstructed from word subunits, but at the expense of recognition accuracy.

[0039] The speech-to-text data store 140 generally serves as a repository for transcripts generated by the speech recognition system 130. In some embodiments, the speech-to-text data store may additionally include a training dataset of words and word subunits, which may be used to train word recognition models and word subunit recognition models used by the speech recognition system 130 to generate transcripts from speech data recordings and to recover out-of-vocabulary words in the generated transcripts, as described above.

[0040] Figure 2 shows that a speech-to-text system (e.g., Figure 1 , which is a block diagram of an example operation performed by a speech recognition system 130 (shown in FIG. 1 ) to generate a transcript of a speech data recording and recover out-of-vocabulary words in the transcript using a word recognition model and a word subunit recognition model. As shown, operation 200 begins at block 210, where the system receives an audio clip to be transcribed. As discussed, the audio clip can be a speech data recording that does not include any other content, an audio track from an audiovisual file, an audiovisual file from which audio content can be extracted, and the like.

[0041] At box 220, the system transcribes the audio clip into a text representation using the trained model. As discussed, the system can use a word recognition model to transcribe the audio clip. The model can be, for example, a connectionist temporal classification model trained against a dataset of whole vocabulary words and various word subunits, such that vocabulary words in the audio clip are mapped to whole words from the dataset, and out-of-vocabulary words in the audio clip are mapped to multiple word subunits to indicate that the corresponding portion of the audio clip does not include the recognized word in the dataset. In another example, the model can be a sequence-to-sequence model trained against a dataset of vocabulary words, such that vocabulary words in the audio clip are mapped to whole words from the dataset, and out-of-vocabulary words are mapped to special or reserved character sequences that indicate that the word cannot be mapped to the corresponding portion of the audio clip.

[0042] At block 230, the system identifies unknown words in the textual representation of the audio clip. As discussed, the system can identify unknown words in the audio clip by identifying a mapping of portions of the audio clip to word sub-units (e.g., phonetic units, syllables, characters, etc.) or to special or reserved character sequences.

[0043] At block 240, for each unknown word identified in the textual representation of the audio clip, the system recovers the unknown word based on the smaller portion of the unknown word in the textual representation of the audio clip. As described above, recovering unknown, out-of-vocabulary words based on smaller recognized portions of the unknown word can be based on the type of model used to identify words in the audio clip. In some embodiments, recovery of unknown words can be based on the concatenation of phonetic or syllable word sub-units and a dictionary that maps these concatenations to possible words or strings. In other embodiments, recovery of unknown words can be based on information of the character timing of characters identified around the timestamp associated with the unknown word in the audio clip.

[0044] At block 250, the system replaces the unknown words in the textual representation of the audio clip with the recovered words.

[0045] At block 260, the system outputs the textual representation of the audio clip including the recovered words. In some embodiments, the system can output the textual representation to a local or remote system for display on a device connected or integrated therewith. In some embodiments, the system can output the textual representation to a data store for persistent storage for future reference.

[0046] Figure 3 An example operation for recovering out-of-vocabulary words in a speech data recording based on a connectionist temporal classification word recognition and word sub-unit recognition model that can be performed by a speech recognition system according to an embodiment is shown. As shown, the operation 300 can begin at block 310, where the speech recognition system identifies a word as an unknown word based on recognition of sub-units of the word by a first language model trained using a dataset of entire words and word sub-units. As discussed, in-vocabulary words can be mapped to entire words or words that appear in a dictionary of words by the first language model (i.e., a word recognition model), while out-of-vocabulary words can be mapped to word sub-units by the first language model. To identify a word as an unknown word, the speech recognition system can identify a sequence of word sub-units between entire words in a transcript of the speech data recording generated by the first language model. The sequence of word sub-units between entire words can be considered to correspond to an out-of-vocabulary word in the speech data recording, and a second language model (i.e., a word sub-unit recognition model) can be used to recover the word for the sequence.

[0047] At block 320, the speech recognition system aggregates (or synthesizes) the sub- units of words generated by the second language model (which can be executed separately or in parallel from the first language model) into a single unit. The second language model can be a word sub-unit recognition unit trained to map portions of the speech data recording to phonetic or syllable units of a word sub-unit, as described above. To identify the sub-units in the transcript of the speech data recording to combine into a single unit corresponding to the unknown word, the system can attempt to identify a starting position of the single unit based on the position of the last whole word identified by the first language model and the next whole word identified by the first language model. The word sub-units between the last whole word and the next whole word can be considered to be components of an out-of-vocabulary word and can be concatenated to generate a single unit for analysis.

[0048] At block 330, the speech recognition system reconstructs the unknown word based on the aggregation of the identified sub-units of words into a single unit and the type of sub-units. For example, where the sub-units identified at block 320 include phonetic representations of words, the speech recognition system can concatenate the strings defining the phonetic representation of each of the plurality of word sub-units into a single phonetic string. The speech-to-text recognition service can use a dictionary and the single unit including the identified sub-units to concatenate the identified sub-units into a phonetic representation and search the dictionary for a definition of any given word identified by the speech recognition system.

[0049] Conversely, where the text-to-speech recognition system is trained to reconstruct previously unknown words using per-syllable recognition, the identified syllables can be combined into a single unit. The speech recognition system can search for information about any given reconstructed word. For example, where the text-to-speech recognition system reconstructs unknown words based on concatenation of syllable word sub-units, the speech recognition system can post-process the concatenated units against a known dictionary to improve the accuracy of the words identified in the speech data recording and output them to a user of the speech recognition system.

[0050] At block 340, the speech recognition system replaces the sub-units of the unknown word with the reconstructed word. The system can replace these sub-units using various string replacement operations. Once replaced, the system can output the text representation of the speech data recording including the recovered word to the client device 120 for display and / or persist the text representation to the speech-to-text data store 140 for future reference.

[0051] Figure 4 Example operations that can be performed by a speech recognition system to recover out-of-vocabulary words in a textual representation of a speech data recording using a first sequence-to-sequence language model configured to recognize entire words in a speech data recording and a second sequence-to-sequence language model configured to recognize individual characters of words in a speech data recording in accordance with an embodiment are shown. As shown, the operations 400 can begin at block 410, where the speech recognition system identifies a word as an unknown word based on a reserved symbol being generated by the sequence-to-sequence word recognition model for the word. As discussed, the word recognition model can map portions of the speech data recording to entire words that exist within a training data set used to train the word recognition model. Accordingly, the word recognition model can map portions of the speech data recording to a special or reserved sequence of characters that indicates that the word is an out-of-vocabulary word.

[0052] At block 420, the speech recognition system derives timing information for the unknown word in the sequence-to-sequence word recognition model and each character timing information generated by the sequence-to-sequence character recognition model. The timing information for the unknown word can be a single attention peak that is recognized between other attention peaks corresponding to recognized or out-of-vocabulary words in the speech data recording. The each character timing information generated by the sequence-to-sequence character recognition model can be used to identify characters that are recognized around the timestamp associated with the attention peak for the unknown word in the word recognition model. Generally, the each character timing information can be used to identify blank or pause characters that separate individual words in a spoken representation of the words, as well as when individual characters are recognized by the word sub-unit recognition model.

[0053] At block 430, the speech recognition system reconstructs the unknown word based on the characters recognized by the sequence-to-sequence character recognition model. As discussed, the recognized characters can be associated with the attention peak between blank characters that bound the attention peak for the unknown word generated by the word recognition system.

[0054] At block 440, the system replaces the reserved symbol with the reconstructed word. By replacing the reserved symbol with the reconstructed word, the speech recognition system can improve the accuracy of a transcript of the audio data generated by the speech recognition system.

[0055] Figure 5 An example system 500 is shown that uses a word recognition model and a word subunit recognition model to identify out-of-vocabulary words in a speech data recording and recover the identified out-of-vocabulary words for inclusion in a text representation of the speech data recording, in accordance with an embodiment. As shown, the system 500 includes, without limitation, a central processing unit 502, one or more I / O device interfaces 505, a network interface 506, a memory 508, a storage device 510, and an interconnect 512, where the I / O device interfaces can allow various I / O devices 514 (e.g., keyboard, display, mouse device, pen input, etc.) to be connected to the system 500.

[0056] The CPU 502 can retrieve and execute programmed instructions stored in the memory 508. Similarly, the CPU 502 can retrieve and store applications that reside in the memory 508. The interconnect 512 transports programmed instructions and application data between the CPU 502, the I / O device interfaces 504, the network interface 506, the memory 508, and the storage device 510. The included CPU 502 represents a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc. Additionally, the included memory 508 represents a random access memory. Further, the storage device 510 can be a disk drive, a solid state drive, a phase change memory device, etc. Although shown as a single unit, the storage device 510 can be a combination of fixed and / or removable storage devices, such as a fixed disk drive, a solid state drive, a removable memory card or optical storage, network attached storage (NAS), or a storage area network (SAN).

[0057] As shown, the memory 508 includes a speech data transcriber 520 and an out-of- vocabulary word restorer 530. The speech data transcriber 520 is generally configured to generate an initial transcript of a speech data recording using a word recognition model that is configured to recognize out-of-vocabulary words as a collection of word subunits or map out-of-vocabulary words in the speech data recording to a sequence of reserved characters that indicate that a word could not be recognized. The out-of-vocabulary word restorer 530 can examine the transcript of the speech data recording generated by the speech data transcriber 520 to identify a sequence of word subunits (e.g., phonetic units, syllables, characters, etc.) or a sequence of reserved characters that includes an indication of an out-of-vocabulary word. Using a combination of word subunits that correspond to an out-of-vocabulary word, the out-of-vocabulary word restorer 530 can generate a replacement string that represents the out-of-vocabulary word and replace the sequence of word subunits or the sequence of reserved characters with the replacement string.

[0058] As shown, the storage device 510 includes a speech-to-text data store 540. The speech-to-text data store 540 is generally an example of a data store in which transcripts of speech data recordings can be stored. In some embodiments, the speech-to-text data store can also store dictionaries used to train word recognition models and word sub-unit recognition models used in identifying and recovering out-of-vocabulary words in speech data recordings, as described above.

[0059] The description of various embodiments of the present application has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0060] Hereinafter, reference is made to the embodiments presented in the present disclosure. However, the scope of the present disclosure is not limited to the embodiments specifically described. Rather, any combination of the following features and elements, whether or not involving different embodiments, is contemplated for enabling and practicing the intended embodiments. Moreover, while the embodiments disclosed herein can achieve advantages over other possible solutions or prior art, whether or not a given embodiment achieves a particular advantage is not limiting to the scope of the present disclosure. Therefore, the following aspects, features, embodiments, and advantages are illustrative only and are not considered elements or limitations of the appended claims unless explicitly recited in a claim. Likewise, reference to "the invention" should not be interpreted as a reference to any one of the inventive subject matter disclosed herein and should not be considered an element or limitation of the appended claims unless explicitly recited in a claim.

[0061] Aspects of the present application can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "circuit," "module" or "system."

[0062] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0063] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or

[0064] The computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions into the respective computing / processing device for storage in a computer readable storage medium within the respective computing / processing device.

[0065] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0066] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0067] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0068] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0069] The computer program product of the present application can be a computer program product which comprises a computer-readable medium bearing computer program code embodied therein for use with a computer. The computer program product can be propagated signal capable of carrying computer program code and can be transmitted by a computer using transmission media and a communication protocol over a communication link, and a medium for making such a computer program available for downloading or for carrying such a downloaded computer program. The computer program product can be a propagated signal (or other transport medium) carrying computer program code.

[0070] While the forgoing is directed to embodiments of the present application, other and further embodiments can be devised without departing from the basic scope of the application, and the scope of the application should be determined from the appended claims.< / unk>

Claims

1. A method for recovering out-of-vocabulary words in a transcription of a speech data recording, comprising: receiving a speech data recording for transcription into a textual representation of the speech data recording; transcribing the speech data recording into the text representation using a word recognition model; identifying unknown words in the text representation; reconstructing the unknown word in the text representation based on identifications of subunits of the unknown word generated by a subunit identification model; modifying the textual representation of the speech data recording by replacing the unknown word with the reconstruction of the unknown word; as well as A modified textual representation of the speech data record is output.

Images