Detecting and recovering out-of-vocabulary words in speech-to-text transcription systems

Abstract

A method for recovering out-of-vocabulary words in a transcription of a speech data recording is provided, comprising: receiving a speech data recording for transcription into a text representation of the speech data recording; transcribing the speech data recording into the text representation using a word recognition model; identifying unknown words in the text representation; reconstructing the unknown words in the text representation based on identification of subunits of the unknown words generated by a subunit recognition model; modifying the text representation of the speech data recording by replacing the unknown words with the reconstruction of the unknown words; and outputting the modified text representation of the speech data recording.

Description

Technical Field

[0001] The present invention relates to speech-to-text transcription systems, and more particularly, to using text recognition models to identify and recover out-of-vocabulary words in speech data. Background Art

[0002] Speech-to-text systems are used in various settings to convert audio content into textual content. For example, a speech-to-text system can be used to generate a transcript of spoken words in real time for future reference or official records (e.g., transcripts of legal proceedings, meetings, etc.). In another example, a speech-to-text system can be used in real time to enhance audio or audiovisual content, such as a live television program, with a textual representation of the audio.

[0003] Typically, a speech-to-text system can be trained to generate a textual representation of audio content by mapping portions of the audio content to whole words. Depending on the size and content of the dictionary used to train the speech-to-text system, it may be difficult for the speech-to-text system to accurately construct a transcript of the audio content. For example, if a speech-to-text system is trained on a small dictionary, a large number of words may not be accurately recognized. In another example, specialized, domain-specific words (e.g., scientific terms) may not be accurately recognized by a speech-to-text system trained using a general knowledge dictionary.

[0004] When a speech-to-text system encounters an out-of-vocabulary word, or a word found in a dictionary used to train the speech-to-text system, the speech-to-text system may output an empty string for the unknown word or attempt to output a known word that sounds like a portion of the unknown word. Outputting an empty string may result in a loss of information between the audio content and the textual representation of the audio content. Meanwhile, outputting a known word that sounds like a portion of the unknown word may result in a meaningless textual representation of the audio content.

[0005] Therefore, there is a need in the art to solve the above problems. Summary of the Invention

[0006] Viewed from a first aspect, the present invention provides a method for recovering out-of-vocabulary words in a transcription of a speech data recording, comprising: receiving a speech data recording for transcription into a text representation of the speech data recording; transcribing the speech data recording into a text representation using a word recognition model; identifying unknown words in the text representation; reconstructing the unknown words in the text representation based on identification of subunits of the unknown words generated by a subunit recognition model; modifying the text representation of the speech data recording by replacing the unknown words with a reconstruction of the unknown words; and outputting the modified text representation of the speech data recording.

[0007] Viewed from another aspect, the present invention provides a system comprising: a processor; and a memory having instructions stored thereon that, when executed by the processor, perform operations for recovering out-of-vocabulary words in a transcription of a speech data recording, the operations comprising: receiving a speech data recording for transcription into a text representation of the speech data recording; transcribing the speech data recording into a text representation using a word recognition model; identifying unknown words in the text representation; reconstructing the unknown words in the text representation based on identification of subunits of the unknown words generated by a subunit recognition model; modifying the text representation of the speech data recording by replacing the unknown words with a reconstruction of the unknown words; and outputting the modified text representation of the speech data recording.

[0008] Viewed from another aspect, the present invention provides a computer program product for recovering out-of-vocabulary words in a transcription of a speech data recording, the computer program product comprising a computer-readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit to perform a method for performing the steps of the present invention.

[0009] Viewed from another aspect, the invention provides a computer program stored on a computer readable medium and loadable into the internal memory of a digital computer, comprising software code portions for performing the steps of the invention when the program is run on the computer.

[0010] Viewed from another aspect, the present invention provides a computer-readable medium having instructions stored thereon that, when executed by a processor, perform operations for recovering out-of-vocabulary words in a transcription of a speech data recording, the operations comprising: receiving a speech data recording for transcription into a text representation of the speech data recording; transcribing the speech data recording into a text representation using a word recognition model; identifying unknown words in the text representation; reconstructing the unknown words in the text representation based on identification of subunits of the unknown words generated by a subunit recognition model; modifying the text representation of the speech data recording by replacing the unknown words with a reconstruction of the unknown words; and outputting the modified text representation of the speech data recording.

[0011] One aspect of the present invention provides a method for recovering out-of-vocabulary words in a transcription of speech data. The method generally includes receiving a speech data recording for transcription into a text representation of the speech data recording. Using a word recognition model, the speech data recording is transcribed into the text representation. Unknown words are identified in the text representation and the unknown words are reconstructed based on identification of subunits of the unknown words generated by a subunit recognition model. The text representation of the speech data recording is modified by replacing the unknown words with the reconstruction of the unknown words, and the modified text representation is output.

[0012] Another aspect of the present disclosure provides a system having a processor and a memory. The memory typically has instructions stored thereon that, when executed by the processor, perform operations for recovering out-of-vocabulary words in a transcription of speech data. The operations generally include receiving a speech data record for transcription into a text representation of the speech data record. Using a word recognition model, transcribing the speech data record into the text representation. Identifying unknown words in the text representation and reconstructing the unknown words based on identification of subunits of the unknown words generated by a subunit recognition model. Modifying the text representation of the speech data record by replacing the unknown words with the reconstruction of the unknown words, and outputting the modified text representation.

[0013] Yet another aspect of the present disclosure provides a computer-readable medium having instructions stored thereon that, when executed by a processor, perform operations for recovering out-of-vocabulary words in a transcription of speech data. The operations generally include receiving a speech data record for transcription into a text representation of the speech data record. Using a word recognition model, transcribing the speech data record into the text representation. Identifying unknown words in the text representation and reconstructing the unknown words based on identification of subunits of the unknown words generated by a subunit recognition model. Modifying the text representation of the speech data record by replacing the unknown words with the reconstruction of the unknown words, and outputting the modified text representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention will now be described, by way of example only, with reference to preferred embodiments, as shown in the accompanying drawings:

[0015] Figure 1 A networked computing environment is shown in which a first text recognition model is used to transcribe audio content into a text representation and a second text recognition model is used to recover unknown words in the text representation, according to an embodiment.

[0016] Figure 2 Example operations for generating a text representation of audio content using a first text recognition model and recovering out-of-vocabulary words in the generated text representation using a second text recognition model are shown, according to an embodiment.

[0017] Figure 3 Example operations for recovering out-of-vocabulary words using a Connectionist Temporal Classification (CTC) text recognition model are shown, according to an embodiment.

[0018] Figure 4 Example operations for recovering out-of-vocabulary words using a sequence-to-sequence text recognition model are shown, according to an embodiment.

[0019] Figure 5 An example system is shown in which aspects of the present disclosure may be performed. DETAILED DESCRIPTION

[0020] Out-of-vocabulary words in audio content often have an adverse effect on the accuracy and readability of the transcripts produced by audio-to-text transcription systems. Because out-of-vocabulary words have a negative impact on transcripts, various techniques have been developed to address the presence of out-of-vocabulary words in audio content. In one example, a hybrid speech recognition system can use various decoding outputs to compensate for the presence of out-of-vocabulary words and attempt to improve the accuracy and readability of the transcript. These outputs may include word confidence information and timing information, which can be used to identify the location and length of out-of-vocabulary words and attempt to recover these words. For example, a speech recognition model can use sub-word units to explicitly model out-of-vocabulary words, or can use confidence scores generated by the speech recognition system to classify text output into out-of-vocabulary and in-vocabulary regions. In other examples, an end-to-end neural network-based speech recognition system can be used to identify and attempt to recover out-of-vocabulary words, using training and inference pipelines that may be simpler than those used in conventional speech recognition systems.

[0021] However, these systems have limitations that reduce the performance of these systems in identifying and recovering out-of-vocabulary words in audio content. Systems that use primarily whole-word based recognition systems and character-based models for out-of-vocabulary recognition and recovery may rely on confidence scores generated by the whole-word based recognition system, which may result in the generation of retain unknown word tokens that ultimately call upon the character-based model for words that may actually be known to the whole-word based recognition system. In addition, these systems may produce inaccurate or no timing information for words in the audio content, which may reduce the performance of out-of-vocabulary word detection in speech recognition systems.

[0022] Aspects of the present disclosure provide efficient techniques for detecting and recovering out-of-vocabulary words in audio content. As discussed in further detail below, out-of-vocabulary words can be detected by determining whether a speech recognition system has reconstructed the word as a whole or as a collection of word fragments (e.g., speech segments, characters, etc.). If the speech recognition system has reconstructed the word as a collection of word fragments, the speech recognition system can consider the word to be an unknown word and attempt to recover the word from the word fragments. By using the determination of whether the word has been reconstructed as a whole word or a collection of word fragments, the aspects discussed herein can bypass the use of confidence information to determine whether a word is in-vocabulary or out-of-vocabulary. The speech recognition system can additionally or alternatively use an entire word sequence-to-sequence system, such as an attention-based model, to identify in-vocabulary and out-of-vocabulary words and reconstruct the out-of-vocabulary word using character output in regions that overlap with the out-of-vocabulary word.

[0023] Figure 1 An example networked computing environment is shown in which whole word and word fragment recognition models are used to identify and recover out-of-vocabulary words in a transcription of audio content according to an embodiment of the present disclosure. As shown, the computing environment 100 includes a client device 120, a speech recognition system 130, and a speech-to-text data storage device 140.

[0024] The client device 120 generally represents a computing device on which a user can access audiovisual content and receive a textual representation of the audiovisual content for display from the speech recognition system 130. As shown, the client device 120 includes a software application 122.

[0025] The software application 122 generally provides an interface through which a user can access audiovisual content from a remote source and invoke speech recognition and transcript generation operations at the speech recognition system 130. In some embodiments, the software application 122 can provide a mechanism for users of the client device 120 to upload previously recorded audiovisual files to the speech recognition system 130 and retrieve transcripts of recorded audiovisual files from it. In some embodiments, the software application 122 can be a web browser or other application that allows a user to interact with the speech recognition system 130.

[0026] The speech recognition service 130 receives audiovisual content and uses word and word subunit (or word fragment) recognition models to generate a transcript of the audiovisual content and outputs the transcript to a client application for display and / or to a data storage device (e.g., speech-to-text data storage device 140) for future retrieval. As shown, the speech recognition system 130 includes a speech data transcriber 132 and an out-of-vocabulary word recoverer 134.

[0027] The speech data transcriber 132 typically processes the received audiovisual content using a word recognition model and a subunit recognition model to generate a textual representation of the audio content, such as a speech data recording or an audiovisual file including a speech data track. The models used by the speech data transcriber 132 can be trained by a remote system (not shown) and deployed to the speech recognition system 130 for use in generating transcripts of the speech data recording, and these models can be periodically updated to improve the accuracy of the transcripts generated using these models. In some embodiments, the word recognition model elements can be trained using a training dataset of whole words and word subunits (such as phonetic units or parts of whole words). As discussed in further detail below, when a word recognition model is trained using a training dataset of whole words and word subunits, a plurality of word subunits that generate a given portion of a speech data recording can be used as an indication that the given portion of the speech data recording corresponds to an out-of-vocabulary word. The word subunit recognition model can be trained using a training dataset of word parts (such as phonetic units, parts of words (e.g., syllables), characters, or other components of words that can be included in a speech data recording). As discussed in further detail below, the output of the word subunit recognition model can be used to recover out-of-vocabulary words.

[0028] The word recognition model and the word subunit recognition model can be implemented as a neural network that is configured to map an audio representation of a word (or portion of a word) to a corresponding textual representation of the word (or portion of a word). In some embodiments, where the word recognition model and the word subunit recognition model are connectionist temporal classification models, the word recognition model can be configured to map a portion of the speech data recording to a word or word subunit, and the subunit recognition model can be configured to map the portion of the speech data recording to one or more phonetic or syllabic word subunits that can be connected with other word subunits to recover the out-of-vocabulary word. In some embodiments, where the word recognition model and the word subunit recognition model are sequence-to-sequence models, the word recognition model can be configured to output timing information and map the portion of the speech data recording to an entire word or to indicate that the word is unknown to the word recognition model, and the word subunit recognition model can be configured to map the portion of the speech data recording to one or more characters and output timing information for each character recognized in the speech data recording.

[0029] The speech data transcriber 132 typically receives a speech data recording as an audio file or an audio track in an audiovisual file and transcribes the speech data recording into a textual representation of the speech data recording. To transcribe the speech data recording into a textual representation, the speech data transcriber 132 may input the speech data recording into a word recognition model that generates a textual output for a given audio input. As described above, the word recognition model may be a neural network configured to map information in the speech data recording to individual whole words or word parts based on, for example, confidence levels associated with mappings of the portions of the speech data recording to words or word parts in the population of words and word parts used to train the word recognition model. For example, the word recognition model may generate a probability distribution over the population of words and word parts used to train the word recognition model and select the word or word part with the highest probability in the probability distribution as the word or word part corresponding to each portion of the speech data recording.

[0030] For a word recognition model that is constructed as a connectionist temporal classification model, the word recognition model can cause a portion of a speech data recording to be mapped to a whole word or word fragment, such as a speech segment, a syllable component of a word, or the like. Generally, mapping a portion of a speech data recording to a whole word can indicate that the word recognition model recognizes that portion of the speech data recording as an in-vocabulary word, while mapping that portion of the speech data recording to a word fragment can indicate that the word recognition model recognizes that portion of the speech data recording (and other adjacent portions of the speech data recording) as out-of-vocabulary words. To determine whether a portion of the speech data recording has been mapped to a whole word or word portion, the speech data transcriber 132 can search a dictionary for an exact match to each word or word portion that is mapped to the portion of the speech data recording. Groupings of contiguous word portions that may not be present in the dictionary can therefore correspond to out-of-vocabulary words for reconstruction by the out-of-vocabulary word restorer 134, as described below.

[0031] For a word recognition model that is constructed as a sequence-to-sequence model, the word recognition model may generate in-vocabulary words for portions of the speech data recording that map to recognized words in the dictionary used to train the sequence-to-sequence model. Conversely, reserved character sequences or reserved word subunits may be generated for portions of the speech data recording that do not map to recognized words in the dictionary. As discussed in further detail below, reserved character sequences or reserved word subunits, such as " <unk>” can indicate to the out-of-vocabulary word restorer 134 the general location of out-of-vocabulary words to be reconstructed from the sequence of individual characters in the speech data recording into the sequence model that is trained to recognize them.

[0032] The out-of-vocabulary word recoverer 134 typically receives a transcript of the speech data recording from the speech data transcriber 132 and uses a corresponding word subunit recognition model to reconstruct the out-of-vocabulary words identified by the word recognition model. The technique used by the out-of-vocabulary word recoverer 134 can differ based on whether the word recognition model and the word subunit recognition model are connectionist temporal classification models, sequence-to-sequence models, or other language recognition models that can be used to transcribe the speech data recording into a textual representation of the speech data recording.

[0033] For a word recognition model configured as a connectionist temporal classification model, each out-of-vocabulary word identified in a transcript of a speech data recording can be defined as a sequence of consecutive word subunits or portions bounded by a previously recognized whole word and a subsequently recognized whole word. To recover out-of-vocabulary words, the out-of-vocabulary word recoverer 134 is typically configured to use a word subunit recognition model that is also configured as a connectionist temporal classification model and is trained to map portions of the speech data recording to word subunits, such as speech segments, word syllables, etc. The out-of-vocabulary word recoverer 134 can identify word subunit segments generated by the word subunit recognition model that correspond to the out-of-vocabulary word in the transcript, connect the word subunits together into one or more word subunit groups, and reconstruct the word from the one or more word subunit groups. In some embodiments where the word subunits are speech segments, the out-of-vocabulary word recoverer 134 can search a reverse speech dictionary to reconstruct the word from each grouping of the speech segments. In other embodiments where the word subunits are syllables or other non-phonetic word subunits, the out-of-vocabulary word recoverer 134 may concatenate the word subunits in the group into a single string and search a dictionary for potential matches to the string. The out-of-vocabulary word recoverer 134 may use fuzzy matching techniques, such as those used in computer-based spelling correction mechanisms, to identify possible matches to the string as words corresponding to the word subunit group. After identifying a word from the word subunit group, the out-of-vocabulary word recoverer 134 may replace the word subunit group in the transcript generated by the speech data transcriber 132 with the identified word, thereby recovering the out-of-vocabulary word in the speech data recording with an accurate representation of the out-of-vocabulary word.

[0034] In embodiments where the word recognition model is a sequence-to-sequence model, out-of-vocabulary words and special character sequences that identify out-of-vocabulary words as unknown words in the transcript generated by the speech data transcriber 132 may be accompanied by attention peaks. The attention peaks may be timestamps in the speech data recording around which the unknown, out-of-vocabulary word is located. Out-of-vocabulary words may be recovered on a per-character basis using a word subunit recognition model, wherein the word subunit recognition model may be a sequence-to-sequence model configured to recognize individual characters in the speech data recording. To identify characters corresponding to out-of-vocabulary words, the out-of-vocabulary word recoverer 134 may use the attention peaks associated with the out-of-vocabulary words to identify possible boundaries of words in the speech data recording provided to the speech recognition system 130 for analysis. Word boundaries may be defined based on the identification of whitespace characters within the speech data recording, wherein the first character of a word is identified immediately after a whitespace character in the speech data recording and the last character of the word is identified immediately before the next whitespace character in the speech data recording. In some embodiments, the surrounding context may also or instead be used to identify characters associated with an out-of-vocabulary word in a speech data recording. For example, information about characters included in words preceding and following an out-of-vocabulary word may be used to identify the boundaries of the out-of-vocabulary word in a character-by-character transcript of the speech data recording generated by a word subunit recognition model.

[0035] Typically, each character recognized in the speech data recorded by the word subunit recognition model may be accompanied by a corresponding attention peak in the model. The out-of-vocabulary word recoverer 134 can generate a string by concatenating the characters associated with each attention peak in the model between whitespace characters or other boundary characters identified for the out-of-vocabulary word. The string generated by the out-of-vocabulary word recoverer 134 can replace the reserved character sequence of the corresponding unknown word in the transcript of the speech data recording.

[0036] After the out-of-vocabulary word restorer 134 modifies the transcript of the speech data recording by replacing the sequence associated with the out-of-vocabulary word with a reconstruction of the out-of-vocabulary word using the word subunit recognition model, the out-of-vocabulary word restorer 134 outputs the modified transcript. In some embodiments, the modified transcript can be output for display in real time or near real time in the software application 122 executing on the client device 120 along with the audiovisual content from which the transcript was generated. In some embodiments, the modified transcript can be submitted to the speech-to-text data storage 140 for future reference.

[0037] In some embodiments, where the speech recognition system 130 uses connectionist temporal classification word recognition and word subunit recognition models, the interpolation weights associated with these models can be adjusted to control the likelihood that a word will be recognized as an out-of-vocabulary word. Higher interpolation weights associated with the word subunit recognition models can increase the number of words that are recognized as out-of-vocabulary words and reconstructed from word subunits (e.g., phonetic units, syllable units, etc.), which can reduce the accuracy of the system while increasing the likelihood that the word can be recalled using the word subunit analysis and reconstruction techniques discussed above. Similarly, lower interpolation weights associated with the word subunit recognition model can reduce the number of words that are recognized as out-of-vocabulary words, which can reduce the number of words that are reconstructed from word subunits, but at the expense of recognition accuracy.

[0038] The speech-to-text data store 140 generally serves as a repository for transcripts generated by the speech recognition system 130. In some embodiments, the speech-to-text data store may additionally include a training dataset of words and word subunits, which may be used to train word recognition models and word subunit recognition models used by the speech recognition system 130 to generate transcripts from speech data recordings and to recover out-of-vocabulary words in the generated transcripts, as described above.

[0039] Figure 2 shows that a speech-to-text system (e.g., Figure 1 , which is a block diagram of an example operation performed by a speech recognition system 130 (shown in FIG. 1 ) to generate a transcript of a speech data recording and recover out-of-vocabulary words in the transcript using a word recognition model and a word subunit recognition model. As shown, operation 200 begins at block 210, where the system receives an audio clip to be transcribed. As discussed, the audio clip can be a speech data recording that does not include any other content, an audio track from an audiovisual file, an audiovisual file from which audio content can be extracted, and the like.

[0040] At box 220, the system transcribes the audio clip into a text representation using the trained model. As discussed, the system can use a word recognition model to transcribe the audio clip. The model can be, for example, a connectionist temporal classification model trained against a dataset of whole vocabulary words and various word subunits, such that vocabulary words in the audio clip are mapped to whole words from the dataset, and out-of-vocabulary words in the audio clip are mapped to multiple word subunits to indicate that the corresponding portion of the audio clip does not include the recognized word in the dataset. In another example, the model can be a sequence-to-sequence model trained against a dataset of vocabulary words, such that vocabulary words in the audio clip are mapped to whole words from the dataset, and out-of-vocabulary words are mapped to special or reserved character sequences that indicate that the word cannot be mapped to the corresponding portion of the audio clip.

[0041] The system identifies unknown words in the text representation of the audio clip at block 230. As discussed, the system can identify unknown words in the audio clip by identifying a mapping of portions of the audio clip to word subunits (e.g., phonetic units, syllables, characters, etc.) or to special or reserved character sequences.

[0042] At block 240, for each unknown word identified in the textual representation of the audio clip, the system recovers the word based on a smaller portion of the unknown word in the textual representation of the audio clip. As described above, recovering unknown, out-of-vocabulary words based on a smaller identified portion of the unknown word can be based on the type of model used to identify words in the audio clip. In some embodiments, recovery of unknown words can be based on connections of phonetic or syllabic word subunits and a dictionary that maps these connections to possible words or character strings. In other embodiments, recovery of unknown words can be based on information about the character timing of characters identified around a timestamp associated with the unknown word in the audio clip.

[0043] At block 250 , the system replaces the unknown word in the textual representation of the audio clip with the recovered word.

[0044] At block 260, the system outputs a text representation of the audio clip including the recovered word. In some embodiments, the system may output the text representation to a local or remote system for display on a device connected or integrated therewith. In some embodiments, the system may output the text representation to a data storage device for persistent storage for future reference.

[0045] Figure 3 Example operations that may be performed by a speech recognition system for recovering out-of-vocabulary words in a speech data recording based on connectionist temporal classification word recognition and word subunit recognition models are shown in accordance with an embodiment. As shown, operations 300 may begin at block 310, where the speech recognition system identifies a word as an unknown word based on recognition of subunits of the word by a first language model trained using a dataset of whole words and word subunits. As discussed, in-vocabulary words may be mapped by the first language model (i.e., the word recognition model) to whole words or words that appear in a dictionary of words, while out-of-vocabulary words may be mapped by the first language model to word subunits. To identify a word as an unknown word, the speech recognition system may identify a sequence of word subunits between whole words in a transcript of the speech data recording generated by the first language model. The sequence of word subunits between whole words may be considered a sequence corresponding to an out-of-vocabulary word in the speech data recording, and the word for the sequence may be recovered using a second language model (i.e., the word subunit recognition model).

[0046] At block 320, the speech recognition system aggregates (or synthesizes) the subunits of the word generated by a second language model (which may be executed separately from or in parallel with the first language model) into a single unit. The second language model may be a word subunit recognition unit trained to map portions of the speech data recording to the phonetic or syllabic units of a word, as described above. To identify subunits in the transcript of the speech data recording to combine into a single unit corresponding to the unknown word, the system may attempt to identify the starting position of the single unit based on the position of the last whole word recognized by the first language model and the next whole word recognized by the first language model. The word subunits between the last whole word and the next whole word may be considered components of an out-of-vocabulary word and may be concatenated to generate a single unit for analysis.

[0047] At block 330, the speech recognition system reconstructs the unknown word based on the aggregation of the identified subunits of the word into a single unit and the type of the subunit. For example, if the subunits identified at block 320 include a phonetic representation of the word, the speech recognition system may concatenate strings defining the phonetic representation of each of the multiple word subunits into a single phonetic string. The speech-to-text recognition service may use a dictionary and the single unit including the identified subunits to concatenate the identified subunits into a phonetic representation and search the dictionary's phonetic representation to identify a definition for any given word identified by the speech recognition system.

[0048] In contrast, where a text-to-speech recognition system is trained to reconstruct previously unknown words using per-syllable recognition, the recognized syllables can be combined into a single unit. The speech recognition system can search for information about any given reconstructed word. For example, where a text-to-speech recognition system reconstructs an unknown word based on the concatenation of syllable-word subunits, the speech recognition system can post-process the concatenated units against a known dictionary to improve the accuracy of the words recognized in the speech data recording and output to the user of the speech recognition system.

[0049] At block 340, the speech recognition system replaces subunits of the unknown word with the reconstructed word. The system may use various string replacement operations to replace these subunits. Once replaced, the system may output a text representation of the speech data recording including the recovered word to the client device 120 for display and / or persist the text representation to the speech-to-text data storage device 140 for future reference.

[0050] Figure 4 Example operations are shown that may be performed by a speech recognition system to recover out-of-vocabulary words in a textual representation of a speech data recording using a first sequence-to-sequence language model configured to recognize entire words in the speech data recording and a second sequence-to-sequence language model configured to recognize individual characters of words in the speech data recording, according to an embodiment. As shown, operation 400 may begin at block 410, where the speech recognition system identifies a word as an unknown word based on reserved symbols generated for the word by the sequence-to-sequence word recognition model. As discussed, the word recognition model may map portions of the speech data recording to entire words present in a training data set used to train the word recognition model. Accordingly, the word recognition model may map portions of the speech data recording to special or reserved character sequences that indicate the word is an out-of-vocabulary word.

[0051] At box 420, the speech recognition system derives timing information for unknown words in the sequence-to-sequence word recognition model and per-character timing information generated by the sequence-to-sequence character recognition model. The timing information for the unknown word can be a single attention peak identified between other attention peaks corresponding to recognized or out-of-vocabulary words in the speech data recording. The per-character timing information generated by the sequence-to-sequence character recognition model can be used to identify characters recognized around the timestamp associated with the attention peak for the unknown word in the word recognition model. Typically, the per-character timing information can be used to identify whitespace or pause characters that separate individual words in the spoken representation of the words, as well as when the word subunit recognition model recognizes individual characters.

[0052] At block 430, the utterance recognition system reconstructs the unknown word based on the characters recognized by the sequence-to-sequence character recognition model.As discussed, the recognized characters may be associated with attention peaks between whitespace characters that bound the attention peaks for the unknown word generated by the word recognition system.

[0053] At block 440, the system replaces the reserved symbols with the reconstructed words.By replacing the reserved symbols with the reconstructed words, the speech recognition system can improve the accuracy of the transcript of the audio data generated by the speech recognition system.

[0054] Figure 5 An example system 500 is shown that uses a word recognition model and a word subunit recognition model to identify out-of-vocabulary words in a speech data recording and recover the identified out-of-vocabulary words for inclusion in a text representation of the speech data recording, according to an embodiment. As shown, the system 500 includes, but is not limited to, a central processing unit 502, one or more I / O device interfaces 505, a network interface 506, a memory 508, a storage device 510, and an interconnect 512, wherein the I / O device interface can allow various I / O devices 514 (e.g., a keyboard, a display, a mouse device, a pen input, etc.) to be connected to the system 500.

[0055] The CPU 502 can retrieve and execute programming instructions stored in the memory 508. Similarly, the CPU 502 can retrieve and store applications residing in the memory 508. Interconnect 512 transmits programming instructions and application data between the CPU 502, the I / O device interface 504, the network interface 506, the memory 508, and the storage device 510. The CPU 502 is included to represent a single CPU, multiple CPUs, a single CPU with multiple processing cores, etc. In addition, the memory 508 is included to represent random access memory. Furthermore, the storage device 510 can be a disk drive, a solid-state drive, a phase-change memory device, etc. Although shown as a single unit, the storage device 510 can be a combination of fixed and / or removable storage devices, such as a fixed disk drive, a solid-state drive, a removable memory card or optical storage, a network attached storage (NAS), or a storage area network (SAN).

[0056] As shown, the memory 508 includes a speech data transcriber 520 and an out-of-vocabulary word recoverer 530. The speech data transcriber 520 is typically configured to generate an initial transcript of the speech data recording using a word recognition model that is configured to recognize out-of-vocabulary words as a collection of word parts or to map out-of-vocabulary words in the speech data recording to a reserved character sequence indicating that the word cannot be recognized. The out-of-vocabulary word recoverer 530 can examine the transcript of the speech data recording generated by the speech data transcriber 520 to identify a sequence of word subunits (e.g., phonetic units, syllables, characters, etc.) or a reserved character sequence that includes an out-of-vocabulary word. Using the combination corresponding to the out-of-vocabulary word subunits, the out-of-vocabulary word recoverer 530 can generate a replacement string representing the out-of-vocabulary word and replace the word subunit or reserved character sequence with the replacement string.

[0057] As shown, storage device 510 includes speech-to-text data storage 540. Speech-to-text data storage 540 is generally an example of a data storage device in which transcripts of speech data recordings may be stored. In some embodiments, speech-to-text data storage device may also store dictionaries used to train word recognition models and word subunit recognition models used in identifying and recovering out-of-vocabulary words in speech data recordings, as described above.

[0058] The description of various embodiments of the present invention has been provided for the purpose of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, practical applications, or improvements over existing technologies in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0059] Below, reference is made to the embodiments presented in the present disclosure. However, the scope of the present disclosure is not limited to the specifically described embodiments. On the contrary, any combination of the following features and elements, whether or not related to different embodiments, is contemplated for use in implementing and practicing the contemplated embodiments. Furthermore, although the embodiments disclosed herein may achieve advantages over other possible solutions or the prior art, whether or not a given embodiment achieves a particular advantage does not limit the scope of the present disclosure. Therefore, the following aspects, features, embodiments and advantages are merely illustrative and are not considered to be elements or limitations of the appended claims unless expressly set forth in the claims. Likewise, reference to "the present invention" should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered to be elements or limitations of the appended claims unless expressly set forth in the claims.

[0060] Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects, which may be collectively referred to herein as a "circuit," "module" or "system."

[0061] The present invention may be a system, method, and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0062] A computer-readable storage medium can be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination thereof. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0063] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0064] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, wherein the programming language comprises object-oriented programming languages ​​(such as Smalltalk, C++ etc.) and conventional procedural programming languages ​​(such as " C " programming languages ​​or similar programming languages). The computer-readable program instructions can be performed completely on the user's computer, partly on the user's computer, performed as an independent software package, partly on the user's computer and partly on a remote computer, or performed completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (such as, using an Internet service provider through the Internet). In certain embodiments, in order to perform various aspects of the present invention, the electronic circuit comprising for example a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can perform the computer-readable program instructions so that the electronic circuit is personalized by utilizing the state information of the computer-readable program instructions.

[0065] Aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowcharts and / or block diagrams and the combination of blocks in the flowcharts and / or block diagrams can be implemented by computer-readable program instructions.

[0066] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can direct the computer, programmable data processing device, and / or other equipment to operate in a specific manner, such that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0067] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0068] Flowchart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in flow chart or block diagram can represent the module, segmentation or part of instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative embodiments, the function mentioned in the frame may not occur in the order mentioned in the figure.For example, the two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, which depends on the function involved.It will also be noted that the combination of the frames in each frame of block diagram and / or flow chart illustration and block diagram and / or flow chart illustration can be realized by the dedicated hardware-based system that performs specified function or action or performs the combination of special-purpose hardware and computer instruction.

[0069] While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope of the invention is determined by the claims hereinafter appended.< / unk>

Claims

1. A method for recovering out-of-vocabulary words in a transcription of a speech data recording, comprising: receiving a speech data recording for transcription into a textual representation of the speech data recording; transcribing the speech data recording into the text representation using a word recognition model, wherein the word recognition model comprises a connectionist temporal classification model trained using a training dataset of whole words and subwords; identifying unknown words in the text representation; reconstructing the unknown word in the text representation based on identifications of subunits of the unknown word generated by a subunit identification model, wherein the subunit identification model comprises a connectionist temporal classification model trained using subwords; modifying the textual representation of the speech data recording by replacing the unknown word with the reconstruction of the unknown word; as well as A modified textual representation of the speech data record is output.

2. The method of claim 1 , wherein the subunits comprise phonetic components of words, and wherein reconstructing the unknown word comprises: combining the identified subunits into a single unit bounded by an end time of a word preceding the unknown word and a start time of a word following the unknown word; as well as The unknown word is looked up using the single unit and a reverse phonetic dictionary.

3. A method according to claim 1, wherein the subunits include fragments from which words can be constructed, each fragment including the beginning of a word symbol and the end of a word symbol, and wherein reconstructing the unknown word includes combining the identified subunits into a single unit by connecting the end of the word symbol of each corresponding subunit to the beginning of the word symbol of a subsequent subunit.

4. A method according to any one of claims 1 to 3, wherein identifying the unknown word in the text representation comprises determining that the word recognition model has output segmented sub-word units in the speech data recording.

5. The method according to any one of claims 1 to 3 further includes adjusting the interpolation weights of the word recognition model to adjust the preference for using whole words or sub-words to generate the text representation.

6. A method for recovering out-of-vocabulary words in a transcription of a speech data recording, comprising: receiving a speech data recording for transcription into a textual representation of the speech data recording; transcribing the speech data recording into the text representation using a word recognition model, wherein the word recognition model comprises a sequence-to-sequence model trained using a training dataset of whole words; identifying unknown words in the text representation; reconstructing the unknown word in the text representation based on identifications of subunits of the unknown word generated by a subunit identification model, wherein the subunit identification model comprises a sequence-to-sequence model trained using a training dataset of subwords; modifying the textual representation of the speech data recording by replacing the unknown word with the reconstruction of the unknown word; as well as A modified textual representation of the speech data record is output.

7. The method of claim 6, wherein identifying the unknown word in the text representation comprises identifying a retained subunit in the text representation, the retained subunit indicating that the word recognition model has identified the unknown word as an out-of-vocabulary word.

8. The method of claim 6 or 7, wherein reconstructing the unknown word comprises: identifying a start time and an end time of the unknown word in the speech data recording based on an attention peak of a whitespace subunit preceding a time peak associated with the unknown word and a whitespace subunit following the time peak associated with the unknown word; as well as Words are constructed based on subunits identified by the subunit identification model at each time peak between the blank space subunit before the time peak associated with the unknown word and the blank space subunit after the time peak associated with the unknown word.

9. A system for recovering out-of-vocabulary words in a transcription of a speech data recording, comprising: processor; as well as a memory having stored thereon instructions that, when executed by the processor, perform operations for recovering out-of-vocabulary words in a transcription of a speech data recording, the operations comprising: receiving a speech data recording for transcription into a textual representation of the speech data recording; transcribing the speech data recording into the text representation using a word recognition model, wherein the word recognition model comprises a connectionist temporal classification model trained using a training dataset of whole words and subwords; identifying unknown words in the text representation; reconstructing the unknown word in the text representation based on identifications of subunits of the unknown word generated by a subunit identification model, wherein the subunit identification model comprises a connectionist temporal classification model trained using subwords; modifying the textual representation of the speech data recording by replacing the unknown word with the reconstruction of the unknown word; and A modified textual representation of the speech data record is output.

10. The system of claim 9, wherein the subunits comprise phonetic components of words, and wherein reconstructing the unknown word comprises: combining the identified subunits into a single unit bounded by an end time of a word preceding the unknown word and a start time of a word following the unknown word; as well as The unknown word is looked up using the single unit and a reverse phonetic dictionary.

11. A system according to claim 9, wherein the subunits include fragments from which words can be constructed, each fragment including the beginning of a word symbol and the end of a word symbol, and wherein reconstructing the unknown word includes combining the identified subunits into a single unit by connecting the end of the word symbol of each corresponding subunit to the beginning of the word symbol of a subsequent subunit.

12. The system of any one of claims 9 to 11, wherein identifying the unknown word in the text representation comprises determining that the word recognition model has output segmented sub-word units in the speech data recording.

13. The system according to any one of claims 9 to 11, wherein: The operations also include adjusting interpolation weights of the word recognition model to adjust a preference for using whole words or sub-words to generate the text representation.

14. A system for recovering out-of-vocabulary words in a transcription of a speech data recording, comprising: processor; as well as a memory having stored thereon instructions that, when executed by the processor, perform operations for recovering out-of-vocabulary words in a transcription of a speech data recording, the operations comprising: receiving a speech data recording for transcription into a textual representation of the speech data recording; transcribing the speech data recording into the text representation using a word recognition model, wherein the word recognition model comprises a sequence-to-sequence model trained using a training dataset of whole words; identifying unknown words in the text representation; reconstructing the unknown word in the text representation based on identifications of subunits of the unknown word generated by a subunit identification model, wherein the subunit identification model comprises a sequence-to-sequence model trained using a training dataset of subwords; modifying the textual representation of the speech data recording by replacing the unknown word with the reconstruction of the unknown word; and A modified textual representation of the speech data record is output.

15. The system of claim 14, wherein identifying the unknown word in the text representation comprises identifying a retained subunit in the text representation, the retained subunit indicating that the word recognition model has identified the unknown word as an out-of-vocabulary word.

16. The system of claim 14 or 15, wherein reconstructing the unknown word comprises: identifying a start time and an end time of the unknown word in the speech data recording based on an attention peak of a whitespace subunit preceding a time peak associated with the unknown word and a whitespace subunit following the time peak associated with the unknown word; as well as Words are constructed based on subunits identified by the subunit identification model at each time peak between the blank space subunit before the time peak associated with the unknown word and the blank space subunit after the time peak associated with the unknown word.

17. A computer program product for recovering out-of-vocabulary words in a transcription of a speech data recording, the computer program product comprising: Instructions for executing the method according to any one of claims 1 to 8.

Images