Method, system and computer program for recovering undefined words in a transcript of a speech data recording
The method and system enhance speech-to-text systems by transcribing audio data, identifying undefined words, and reconstructing them using subunit recognition, improving transcript accuracy and readability.
Patent Information
- Application Number
- JP2022509091
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-13
- Filing Date
- 2020-09-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-09-10
AI Technical Summary
Speech-to-text systems struggle to accurately recognize undefined words, leading to information loss or generation of meaningless textual representations due to outputting null strings or known words that sound like parts of unknown words.
A method and system that utilize a word recognition model to transcribe audio data, identify unknown words, and reconstruct them using a subunit recognition model, replacing undefined words with reconstructions based on subunit recognition.
Improves the accuracy and readability of transcripts by effectively recovering undefined words, enhancing the reliability of speech-to-text systems.
Smart Images

Figure 0007593721000001 
Figure 0007593721000002 
Figure 0007593721000003
Abstract
Description
[Technical field]
[0001] The present invention relates to speech-to-text transcription systems, and more particularly to identifying and recovering undefined words in speech data using text recognition models. [Background technology]
[0002] Speech-to-text systems are used in a variety of devices to convert audio content into text content. For example, speech-to-text systems can be used to generate real-time transcripts of speech for future reference or official records (e.g., transcripts of legal proceedings, meetings, etc.). In another example, speech-to-text systems can be used to supplement audio or audiovisual content, such as a live television program, in real time with a textual representation of the speech.
[0003] In general, speech-to-text systems can be trained to generate textual representations by mapping portions of speech content to whole words. Depending on the size and content of the dictionary used to train the speech-to-text system, it may be difficult for the speech-to-text system to accurately construct a transcript of the speech content. For example, if the speech-to-text system is trained with a small dictionary, many words may not be recognized accurately. In another example, specialized domain-specific words (e.g., scientific terms) may not be recognized accurately by a speech-to-text system trained with a general knowledge dictionary.
[0004] When a speech-to-text system encounters an out-of-vocabulary word, i.e., a word that is not found in the dictionary used to train the speech-to-text system, the speech-to-text system either outputs a null string for the unknown word, or attempts to output a known word that sounds like part of the unknown word. Outputting a null string can result in a loss of information between the speech content and the textual representation of the speech content. On the other hand, outputting a known word that sounds like part of an unknown word can result in the production of a meaningless textual representation of the speech content.
[0005] Therefore, there is a need in the art to address the above-mentioned problems. Summary of the Invention
[0006] According to a first aspect, the present invention provides a method for recovering undefined words in transcriptions of speech data recordings, the method comprising receiving the speech data recording for transcription into a text representation of the speech data recording, transcribing the speech data recording into the text representation using a word recognition model, identifying unknown words in the text representation, reconstructing the unknown words in the text representation based on recognition of subunits of the unknown words generated by a subunit recognition model, modifying the text representation of the speech data recording by replacing the unknown words with reconstructions of the unknown words, and outputting a modified text representation of the speech data recording.
[0007] According to yet another aspect, the invention provides a system comprising a processor and a memory having stored thereon instructions which, when executed by the processor, perform operations for recovering undefined words in a transcript of a speech data recording, the operations including: receiving a speech data record for transcription of the speech data recording into a text representation; transcribing the speech data recording into the text representation using a word recognition model; identifying unknown words in the text representation; reconstructing unknown words in the text representation based on recognition of subunits of the unknown words generated by the subunit recognition model; modifying the text representation of the speech data recording by replacing the unknown words with reconstructions of the unknown words; and outputting the modified text representation of the speech data recording.
[0008] According to yet another aspect, the present invention provides a computer program product for recovering undefined words in a transcript of an audio data recording, the computer program product comprising a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit to perform a method performing steps of the present invention.
[0009] According to yet another aspect, the present invention provides a computer program stored on a computer readable medium and loadable into an internal memory of a digital computer, the computer program comprising software code portions for performing the steps of the present invention when the program is run on the computer.
[0010] According to yet another aspect, the invention provides a computer readable medium having instructions stored thereon that, when executed by a processor, perform operations to restore undefined words in a transcription of a speech data recording, the operations including receiving a speech data recording for transcription into a text representation of the speech data recording, transcribing the speech data recording into the text representation using a word recognition model, identifying unknown words in the text representation, reconstructing the unknown words in the text representation based on recognition of subunits of the unknown words generated by the subunit recognition model, modifying the text representation of the speech data recording by replacing the unknown words with reconstructions of the unknown words, and outputting the modified text representation of the speech data recording.
[0011] One aspect of the disclosure provides a method for recovering undefined words in a transcript of speech data. The method generally includes receiving a speech data record for transcription into a text representation of the speech data record. The speech data record is transcribed into the text representation using a word recognition model. Unknown words in the text representation are identified and the unknown words are reconstructed based on recognition of subunits of the unknown words generated by a subunit recognition model. The text representation of the speech data record is modified by replacing the unknown words with reconstructions of the unknown words and a modified text representation is output.
[0012] Another aspect of the disclosure provides a system having a processor and a memory. The memory generally stores instructions that, when executed by the processor, perform operations to restore undefined words in a transcript of speech data. The operations generally include receiving a speech data record for transcription of the speech data record into a text representation. The speech data record is transcribed into the text representation using a word recognition model. Unknown words are identified in the text representation and the unknown words are reconstructed based on recognition of subunits of the unknown words generated by the subunit recognition model. The text representation of the speech data record is modified by replacing the unknown words with reconstructions of the unknown words and the modified text representation is output.
[0013] Yet another aspect of the present disclosure provides a computer-readable medium having stored thereon instructions that, when executed by a processor, perform operations for recovering undefined words in a transcript of speech data. The operations generally include receiving a speech data recording for transcription of the speech data recording into a text representation. The speech data recording is transcribed into the text representation using a word recognition model. Unknown words are identified in the text representation and the unknown words are reconstructed based on recognition of subunits of the unknown words generated by the subunit recognition model. The text representation of the speech data recording is modified by replacing the unknown words with reconstructions of the unknown words and the modified text representation is output.
[0014] The invention will now be described, by way of example only, with reference to preferred embodiments illustrated in the following drawings. [Brief description of the drawings]
[0015] [Figure 1] 1 illustrates a networked computing environment in which speech content is transcribed into a text representation using a first text recognition model and unknown words in the text representation are recovered using a second text recognition model, according to one embodiment. [Diagram 2] 1 illustrates example operations for generating a text representation of audio content using a first text recognition model and recovering undefined words in the generated text representation using a second text recognition model, according to one embodiment. [Diagram 3] 1 illustrates an exemplary operation for recovering an undefined word using a Connectionist Temporal Classification (CTC) text recognition model, according to one embodiment. [Figure 4] 1 illustrates an example operation for recovering an undefined word using a sequence-to-sequence text recognition model, according to one embodiment. [Diagram 5] 1 illustrates an exemplary system in which aspects of the present disclosure can be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Undefined words in audio content generally adversely affect the accuracy and readability of transcripts generated by speech-to-text transcription systems. Because undefined words adversely affect transcripts, various techniques have been developed to address undefined words in audio content. In one example, a hybrid speech recognition system can use various decoded outputs to compensate for the presence of undefined words and to attempt to improve the accuracy and readability of the transcript. These outputs can include word confidence information and timing information, which can be used to identify the location and length of undefined words and attempt to recover these words. For example, a speech recognition model can explicitly model undefined words using subword units, or can use confidence scores generated by the speech recognition system to classify text output into undefined and in-vocabulary regions. In another example, an end-to-end neural network-based speech recognition system, which uses a training and inference pipeline that can be simpler than those used in traditional speech recognition systems, can be used to identify undefined words and attempt to recover them.
[0017] However, these systems have limitations that reduce their performance in identifying and recovering undefined words in audio content. Systems that use an initial whole-word-based recognition system and a character-based model for undefined recognition and recovery may rely on confidence scores generated by the whole-word-based recognition system, which may generate reserved unknown word tokens that ultimately invoke character-based models of words that may actually be known to the whole-word-based recognition system. Furthermore, these systems may generate inaccurate or no timing information for words in audio content, which may reduce the performance of undefined words in the speech recognition system.
[0018] Aspects of the present disclosure provide efficient techniques for detecting and recovering undefined words in audio content. As described in more detail below, undefined words can be detected by determining whether the speech recognition system reconstructs whole words or a collection of word fragments (e.g., speech segments, characters, etc.). If the speech recognition system reconstructs a word as a collection of word fragments, the speech recognition system can consider the word to be an unknown word and attempt to recover the word from the word fragments. Using the determination of whether a word was reconstructed as a whole word or a collection of word fragments, aspects described herein can bypass the use of confidence information to determine whether a word is defined or undefined. The speech recognition system can additionally or alternatively use a whole word sequence-to-sequence system, such as an attention-based model, to identify defined and undefined words and reconstruct the undefined word using the character output in the area overlapping the undefined word.
[0019] 1 illustrates an exemplary networked computing environment in which a whole word or word fragment recognition module is used to identify and recover undefined words in a transcript of audio content, according to one embodiment of the present disclosure. As shown, the computing environment 100 includes a client device 120, a speech recognition system 130, and a speech-to-text data store 140.
[0020] Client device 120 is a general representation of a computing device through which a user can access audiovisual content and receive a textual representation of the audiovisual content for display from speech recognition system 130. As shown, client device 120 includes software application 122.
[0021] Software application 122 generally provides an interface through which a user can access audio-image content from remote sources and invoke speech recognition and transcript generation operations in speech recognition system 130. In some embodiments, software application 122 provides a mechanism for a user of client device 120 to upload previously recorded audio-image files to speech recognition system 130 and to obtain transcripts of recorded audio-image files from speech recognition system 130. In some embodiments, software application 122 can be a web browser or other application that allows a user to interact with speech recognition system 130.
[0022] The speech recognition system 130 receives the audiovisual content, uses word and word subunit (or word fragment) recognition models to generate a transcript of the audiovisual content, and outputs the transcript to a client application for display or to a data store (e.g., speech-to-text data store 140) for future retrieval, or both. As shown, the speech recognition system 130 includes a speech data transcriber 132 and an undefined word restorer 134.
[0023] The audio data transcriber 132 generally processes the received audio-image content through word recognition models and subunit recognition models to generate a textual representation of the audio content, such as an audio data recording or an audio-image file that includes an audio data track. The models used by the audio data transcriber 132 may be trained by a remote system (not shown) and deployed to the audio recognition system 130 for use in generating transcripts of the audio data recordings, and the models may be periodically updated to improve the accuracy of the transcripts generated using the models. In some embodiments, the word recognition models may be trained using a training data set of whole words and word subunits, such as phonetic units or portions of whole words. As described in more detail below, when the word recognition models are trained using a training data set of whole words and word subunits, the generation of multiple word subunits for a given portion of the audio data recording may be used as an indication that the given portion of the audio data recording corresponds to an undefined word. The word subunit recognition models may be trained using a training data set of word portions, such as phonetic units of words, parts of words (e.g., syllables), characters, or other components that may be included in the audio data recording. The output of the word subunit recognition model can be used to recover undefined words, as described in more detail below.
[0024] The word recognition model and the word subunit recognition model may be implemented as neural networks configured to map phonetic representations of words (or parts of words) to textual representations of corresponding words (or parts of words). In some embodiments where the word recognition model and the word subunit recognition model are connectionist temporal classification models, the word recognition model may be configured to map portions of the speech data record to words or word subunits, and the subunit recognition model may be configured to map portions of the speech data record to one or more phonetic or syllabic word subunits that can be concatenated with other word subunits to recover undefined words. In some embodiments where the word recognition model and the word subunit recognition model are sequence-to-sequence models, the word recognition model may be configured to output timing information and map portions of the speech data record to whole words or reserved character strings to indicate that the word is unknown to the word recognition model, and the word subunit recognition model may be configured to map portions of the speech data record to one or more characters and output timing information for each character recognized in the speech data record.
[0025] The audio data transcriber 132 typically receives an audio data recording as an audio track in an audio image file and transcribes the audio data recording into a textual representation of the audio data recording. To transcribe the audio data recording into a textual representation, the audio data transcriber 132 may input the audio data recording into a word recognition model that generates a textual output for a given audio input. As described above, the word recognition model may be, for example, a neural network configured to map information in the audio data record to individual whole words or word portions based on a confidence associated with mapping portions of the audio data recording to words or word portions in a population of words and word portions used to train the word recognition model. For example, the word recognition model may generate a probability distribution over the population of words and word portions used to train the word recognition model and select the word or word portion with the greatest probability in the probability distribution as the word or word portion corresponding to each portion of the audio data recording.
[0026] For a word recognition model configured as a connectionist temporal classification model, the word recognition model may result in a portion of the speech data recording being mapped to whole words or word fragments, such as phonetic segments of words, syllabic components, etc. In general, a mapping of a portion of the speech data recording to a whole word may indicate that the word recognition model has recognized the portion of the speech data recording as a defined word, whereas a mapping of a portion of the speech data recording to a word fragment may indicate that the word recognition model has recognized the portion of the speech data recording (and other adjacent portions of the speech data recording) as an undefined word. To determine whether the portion of the speech data recording is mapped to a whole word or a word fragment, the speech data transcriber 132 searches a dictionary for an exact match of each word or word fragment mapped to the portion of the speech data recording. Groupings of consecutive word fragments that may not be found in the dictionary may thus correspond to undefined words that the undefined word restorer 134 reconstructs, as described below.
[0027] For a word recognition model configured as a sequence-to-sequence model, the word recognition model may generate predefined words for portions of the speech data recording that map to recognized words in the dictionary used to train the sequence-to-sequence model. In contrast, for portions of the speech data recording that do not map to recognized words in the dictionary, the word recognition model may generate reserved strings or reserved word subunits. A reserved string or reserved word subunit, such as “ <unk>" can indicate to the undefined word restorer 134 the general location of the undefined word to be reconstructed from a sequence-to-sequence model trained to recognize individual characters in the speech data recording, as described in more detail below.
[0028] The undefined word restorer 134 generally receives the transcript of the speech data recording from the speech data transcriber 132 and reconstructs the undefined words identified by the word recognition model using corresponding word subunit recognition models. The techniques used by the undefined word restorer 134 may vary based on whether the word recognition model and the word subunit model are connectionist temporal classification models, sequence-to-sequence models, or other language recognition models that can be used to transcribe the speech data recording into a textual representation of the speech data recording.
[0029] For a word recognition model configured as a connectionist temporal classification model, each undefined word identified in the transcript of the speech data record can be defined as a sequence of contiguous word subunits or parts bounded by the entire immediately preceding and entire immediately succeeding recognized words. To reconstruct the undefined words, the undefined word reconstructor 134 is generally configured to use a word subunit recognition model also configured as a connectionist temporal classification model and trained to map portions of the speech data record to word subunits such as speech segments, word syllables, etc. The undefined word reconstructor 134 can identify word subunit segments generated by the word subunit recognition model that correspond to the undefined words in the transcript, concatenate the word subunits into one or more groups of word subunits, and reconstruct words from one or more groups of word subunits. In some embodiments where the word subunits are speech segments, the undefined word reconstructor 134 can search a reverse speech dictionary to reconstruct words from each group of speech segments. In other embodiments where the word subunits are syllables or other non-speech word subunits, the undefined word restorer 134 may concatenate the word subunits in the group into a single string and search a dictionary for possible matches to the string. The undefined word restorer 134 may use fuzzy matching techniques, such as those used in computer-based spelling correction mechanisms, to identify likely matches to the string as words corresponding to the group of word subunits. After identifying a word from the group of word subunits, the undefined word restorer 134 may replace the group of word subunits in the transcript produced by the speech data transcriber 132 with the identified word, thus restoring the undefined word in the speech data recording with an accurate representation of the undefined word.
[0030] In embodiments where the word recognition model is a sequence-to-sequence model, undefined words and special character strings that identify the undefined words as unknown words in the transcripts generated by the audio data transcriber 132 may be associated with attention peaks. The attention peaks may be timestamps of the audio data recordings where the unknown undefined words are located. The word subunit recognition model may be a sequence-to-sequence model configured to recognize individual characters in the audio data recordings and may be used to recover the undefined words on a character-by-character basis. To identify characters that correspond to undefined words, the undefined word recoverer 134 may use the attention peaks associated with the undefined words to identify likely boundaries of words in the audio data records that are provided to the speech recognition system 130 for analysis. The word boundaries may be determined based on identifying whitespace characters in the audio data records, where the first character of a word is identified immediately after a whitespace character and the last character of a word is identified immediately before the next whitespace character in the audio data records. In some embodiments, surrounding context may also or alternatively be used to identify characters associated with undefined words in the audio data records. For example, information about the characters contained in the words immediately preceding and following an undefined word can be used to identify undefined word boundaries within a character-by-character transcription of a speech data recording produced by a word subunit recognition model.
[0031] In general, each character recognized in the speech data recording by the word subunit recognition model may be associated with a corresponding attention peak in the model. The undefined word restorer 134 may generate a string by concatenating the characters associated with each attention peak in the model between spaces or other boundary characters identified for the undefined word. The string generated by the undefined word restorer 134 may replace the reserved string corresponding to the unknown word in the transcript of the speech data recording.
[0032] After the undefined word restorer 134 modifies the transcript of the speech data recording by replacing sequences associated with the undefined words with reconstructions of the undefined words using the word subunit recognition model, the undefined word restorer 134 outputs the modified transcript. In some embodiments, the modified transcript may be output for display in real-time or near real-time in the software application 122 running on the client device 120 along with the audiovisual content from which the transcript was generated. In some embodiments, the modified transcript may be passed to the audio-text data store 140 for future reference.
[0033] In some embodiments in which the speech recognition system 130 uses connectionist temporal classification word recognition models and word subunit recognition models, the interpolation weights associated with these models can be adjusted to control the likelihood that a word will be recognized as an undefined word. Higher interpolation weights associated with the word subunit recognition models can increase the number of words that are recognized as undefined words and reconstructed from word subunits (e.g., phonetic units, syllable units, etc.), thereby decreasing the accuracy of the system but increasing the likelihood that the word can be recalled using the word subunit analysis and reconstruction techniques described above. Similarly, lowering the interpolation weights associated with the word subunit recognition models can decrease the number of words that are recognized as undefined words, thereby decreasing the number of times a word is reconstructed from word subunits, at the expense of recognition accuracy.
[0034] The audio-to-text data store 140 generally serves as a repository of transcripts generated by the speech recognition system 130. In some embodiments, the audio-to-text data store may additionally include training data sets of words and word subunits that may be used to train word and word subunit recognition models used by the speech recognition system 130 to generate transcripts from speech data recordings and to recover undefined words in the generated transcripts, as described above.
[0035] 2 illustrates exemplary operations that may be performed by a speech-to-text system (e.g., speech recognition system 130 shown in FIG. 1) to generate a transcript of an audio data recording and recover undefined words in the transcript using word and word subunit recognition models, according to one embodiment. As shown, operation 200 begins at block 210, where the system receives an audio clip to be transcribed. As previously discussed, the audio clip may be an audio data recording that does not contain any other content, an audio track from an audio-image file, an audio-image file from which audio content can be extracted, etc.
[0036] At block 220, the system transcribes the audio clip into a textual representation using a trained model. As previously described, the system may transcribe the audio clip using a word recognition model. The model may be, for example, a connectionist temporal classification model trained on a data set of all predefined words and various word subunits, whereby predefined words in the audio clip are mapped to entire words from the data set, while undefined words in the audio clip are mapped to multiple word subunits to indicate that the corresponding portion of the audio clip does not contain a recognized word in the data set. In another example, the model may be a sequence-to-sequence model trained on a data set of predefined words, whereby predefined words in the audio clip are mapped to entire words in the data set, and undefined words are mapped to special or reserved strings to indicate that the word could not be mapped to the corresponding portion of the audio clip.
[0037] At block 230, the system identifies unknown words in the textual representation of the audio clip. As discussed above, the system can identify unknown words in the audio clip by identifying either a mapping of portions of the audio clip to word subunits (e.g., phonetic units, syllables, characters, etc.) or a mapping of portions of the audio clip to special or reserved strings of characters.
[0038] At block 240, for each unknown word identified in the textual representation of the audio clip, the system reconstructs the word based on smaller portions of the unknown word in the textual representation of the audio clip. Reconstruction of the unknown undefined word based on smaller recognized portions of the unknown word may be based on the type of model used to recognize the word in the audio clip, as described above. In some embodiments, reconstruction of the unknown word may be based on concatenations of phonetic or syllabic word subunits and a dictionary that maps these concatenations to likely words or strings of characters. In other embodiments, reconstruction of the unknown word may be based on timing information about characters recognized around a timestamp associated with the unknown word in the audio clip.
[0039] At block 250, the system replaces the unknown words in the textual representation of the audio clip with the recovered words.
[0040] At block 260, the system outputs a textual representation of the audio clip containing the recovered words. In some embodiments, the textual representation may be output to a local or remote system for display on a device connected to or integrated with the system. In some embodiments, the system may output the textual representation to persist in a data store for future reference.
[0041] FIG. 3 illustrates exemplary operations that may be performed by a speech recognition system to recover undefined words in a speech data recording using a connectionist temporal classification word recognition model and a word subunit recognition model, according to one embodiment. As illustrated, operations 300 begin at block 310, where the speech recognition system identifies a word as an unknown word based on recognition of subunits of the word by a first language model trained with a data set of whole words and word subunits. As previously described, the first language model (e.g., the word recognition model) may map predefined words to whole words or words appearing in a dictionary of words, and the first language model may map undefined words to word subunits. To identify a word as an unknown word, the speech recognition system may identify sequences of word subunits among whole words in a transcript of the speech data recording generated by the first language model. Sequences of word subunits located between whole words can be considered as sequences corresponding to undefined words in the speech data recording, and words can be reconstructed for the sequences using a second language model (e.g., a word subunit recognition model).
[0042] In block 320, the speech recognition system aggregates (or synthesizes) the word subunits generated by the second language model (which may run separately or in parallel with the first language model) into a single unit. The second language model may be a word subunit recognition model trained to map portions of the speech data record to word phonetic or syllabic units, as described above. To identify and combine the subunits in the transcript of the speech data record into a single unit corresponding to the unknown word, the system may attempt to identify the start location of the single unit based on the location of the last whole word recognized by the first language model and the next whole word recognized by the first language model. Word subunits between the last whole word and the next whole word may be considered components of the undefined word and may be concatenated to generate a single unit for analysis.
[0043] In block 330, the speech recognition system reconstructs the unknown word based on the aggregation of the recognized word subunits into a single unit and the type of the subunit. For example, if the subunits recognized in block 320 include a phonetic representation of a word, the speech recognition system may concatenate the strings defining the phonetic representations of each of the multiple word subunits into a single phonetic string. The speech-to-text recognition system may use a dictionary and the single unit that includes the recognized subunits to concatenate the recognized subunits into the phonetic representation and look up the phonetic representation in the dictionary to identify a definition of any given word recognized by the speech recognition system.
[0044] In contrast, if a speech-to-text recognition system has been previously trained to reconstruct unknown words using syllable-by-syllable recognition, the recognized syllables can be combined into a single unit. The speech recognition system can retrieve information about any given reconstructed word. For example, if the speech-to-text recognition system reconstructs unknown words based on concatenation of syllabic word subunits, the speech recognition system can post-process the concatenated units against a known dictionary to improve the accuracy of the words recognized in the speech data recording and output them to a user of the speech recognition system.
[0045] In block 340, the speech recognition system replaces subunits of the unknown word with reconstructed words. The system may replace these subunits using a variety of string replacement operations. Once replaced, the system may output a text representation of the speech data recording, including the reconstructed words, to the client device 120 for display, or may persist the text representation in a speech-to-text data store for future reference, or may do both.
[0046] FIG. 4 illustrates exemplary operations that may be performed by a speech recognition system to recover undefined words in a textual representation of a speech data recording using a first sequence-to-sequence language model configured to recognize entire words in the speech data recording and a second sequence-to-sequence language model configured to recognize individual characters of words in the speech data recording, according to one embodiment. As shown, operations 400 begin at block 410, where the speech recognition system identifies a word as an unknown word based on the generation of reserved symbols for the word by the sequence-to-sequence word recognition model. As previously described, the word recognition model may map portions of the speech data recording to entire words present in a training data set used to train the word recognition model. Similarly, the word recognition model may map portions of the speech data recording to special or reserved character strings to indicate that the word is an undefined word.
[0047] In block 420, the speech recognition system obtains timing information for unknown words in the sequence-to-sequence word recognition model and per-character timing information generated by the sequence-to-sequence character recognition model. The timing information for the unknown word may be a single attention peak identified among other attention peaks corresponding to recognized or undefined words in the speech data recording. The per-character timing information generated by the sequence-to-sequence character recognition model may be used to identify characters recognized around a timestamp associated with the attention peak for the unknown word in the word recognition model. In general, the per-character timing information may be used to identify space or pause characters that separate individual words in the phonetic representation of these words and when individual characters are recognized by the word subunit recognition model.
[0048] In block 430, the speech recognition system reconstructs the unknown word based on the characters recognized by the sequence-to-sequence character recognition model, which may be associated with attention peaks between whitespace characters that indicate boundaries of attention peaks of the unknown word generated by the word recognition system, as described above.
[0049] The system replaces the reserved symbols with the reconstructed words at block 440. By replacing the reserved symbols with the reconstructed words, the speech recognition system can improve the accuracy of the transcriptions of the speech data produced by the speech recognition system.
[0050] 5 illustrates an exemplary system 500 for identifying undefined words in an audio data recording and recovering the identified undefined words for inclusion in a textual representation of the audio data recording using a word recognition model and a word subunit recognition model, according to one embodiment. As shown, system 500 includes, without limitation, a central processing unit 502, one or more I / O device interfaces 504 that enable connection of various I / O devices 514 (e.g., keyboard, display, mouse device, pen input, etc.) to system 500, a network interface 506, memory 508, storage 510, and an interconnect 512.
[0051] CPU 502 can read and execute programming instructions stored in memory 508. Similarly, CPU 502 can read and store applications resident in memory 508. Interconnect 512 routes programming instructions and application data between CPU 502, I / O device interface 504, network interface 506, memory 508, and storage 510. CPU 502 is included as representative of a single CPU, multiple CPUs, a single CPU with multiple processing cores, and the like. Additionally, memory 508 is included as representative of random access memory. Additionally, storage 510 can be a disk drive, solid state drive, phase change memory device, and the like. Although shown as a single unit, storage 510 can be a combination of fixed and / or removable storage devices, such as a fixed disk drive, solid state drive, removable memory card or optical storage, network attached storage (NAS), or storage area network (SAN), and the like.
[0052] As shown, the memory 508 includes an audio data transcriber 520 and an undefined word restorer 530. The audio data transcriber 520 is generally configured to generate an initial transcription of the audio data recording using a word recognition model configured to recognize undefined words as a collection of word parts or to map undefined words in the audio data recording to reserved strings to indicate that the word could not be recognized. The undefined word restorer 530 examines the transcription of the audio data recording generated by the audio data transcriber 520 to identify the inclusion of a sequence of word subunits (e.g., speech units, syllables, characters, etc.) or reserved strings that indicate an undefined word. Using a combination of word subunits that corresponds to the undefined word, the undefined word restorer 530 generates a replacement string that represents the undefined word and replaces the word subunits or reserved strings with the replacement string.
[0053] Storage 510, as shown, includes speech-to-text data store 540. Speech-to-text data store 540 is a general example of a data store that may store transcripts of speech data recordings. In some embodiments, the speech-to-text data store may also store dictionaries used to train word recognition models and word subunit recognition models used to identify and recover undefined words in the speech data recordings, as described above.
[0054] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but they are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.
[0055] In the following, reference is made to the embodiments presented in this disclosure. However, the scope of the disclosure is not limited to the specific embodiments described. Instead, any combination of the following features and elements, whether related to different embodiments or not, is contemplated to implement and practice the contemplated embodiments. Furthermore, although the embodiments disclosed herein may realize advantages over other possible solutions or over the prior art, whether or not a particular advantage is realized by a given embodiment does not limit the scope of the disclosure. Thus, the following aspects, features, embodiments, and advantages are merely illustrative and should not be considered as elements or limitations of the appended claims unless expressly recited in the claims. Similarly, references to the "invention" should not be construed as a generalization of the subject matter of any invention disclosed herein, and should not be considered as elements or limitations of the appended claims unless expressly recited in the claims.
[0056] Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all collectively referred to herein as a "circuit," "module," or "system."
[0057] The present invention may be a system, method, or computer program product, or a combination thereof. The computer program product may include a computer readable storage medium or media having computer readable program instructions thereon for causing a processor to perform aspects of the present invention.
[0058] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical coding devices such as punch cards or raised groove structures having instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium is not to be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or an electrical signal sent through a wire.
[0059] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.
[0060] The computer readable program instructions for carrying out the operations of the present disclosure may be source or object code written in any combination of assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or one or more programming languages, including object oriented programming languages such as Smalltalk, C++, and traditional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the last scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.
[0061] Aspects of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.
[0062] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer program instructions may also be stored in a computer readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the instructions stored in the computer readable storage medium include an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.
[0063] The computer program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.
[0064] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions shown in the blocks may be performed in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or these blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that executes the specified functions or operations, or a combination of dedicated hardware and computer instructions.
[0065] While the forgoing is directed to embodiments of the present disclosure, other and further embodiments of the invention may be devised without departing from the basic scope thereof, which scope is determined by the following claims.< / unk>
Claims
1. 1. A method for recovering undefined words in a transcript of an audio data recording, comprising: receiving an audio data recording for transcription into a text representation of the audio data recording; transcribing the speech data recording into the text representation using a word recognition model, the word recognition model including a connectionist temporal classification model trained using a training data set of whole words and word subunits; identifying unknown words in the text representation; reconstructing the unknown words in the text representation based on recognition of subunits of the unknown words generated by a subunit recognition model including a connectionist temporal classification model trained with word subunits; modifying the text representation of the speech data recording by replacing the unknown words with the reconstructions of the unknown words; outputting a modified text representation of the audio data recording; and The method includes:
2. the subunits include phonetic components of words; Reconstructing the unknown word comprises: consolidating the recognized subunits into a single unit bounded by an end time of a word immediately preceding the unknown word and a start time of a word immediately following the unknown word; looking up the unknown word using the single unit and a reverse phonetic dictionary; Including, The method of claim 1.
3. 2. The method of claim 1, wherein the subunits include fragments from which words can be constructed, each fragment including a word-start symbol and a word-end symbol, and reconstructing the unknown word includes combining the recognized subunits into a single unit by concatenating the word-end symbol of each subunit to the word-start symbol of the immediately following subunit.
4. 4. The method of claim 1, wherein identifying the unknown words in the text representation comprises determining that the word recognition model has output sub-word units for segments in the speech data recording.
5. The method of any one of claims 1 to 4, further comprising adjusting interpolation weights of the word recognition model to adjust a preference for using whole words or subwords to generate the text representation.
6. A processor; a memory having instructions stored thereon that, when executed by said processor, perform operations to recover undefined words in a transcript of an audio data recording; Equipped with The operation comprises: receiving an audio data recording for transcription into a text representation of the audio data recording; transcribing the speech data recording into the text representation using a word recognition model, the word recognition model including a connectionist temporal classification model trained using a training data set of whole words and word subunits; identifying unknown words in the text representation; reconstructing the unknown words in the text representation based on recognition of subunits of the unknown words generated by a subunit recognition model including a connectionist temporal classification model trained with word subunits; modifying the text representation of the speech data recording by replacing the unknown words with the reconstructions of the unknown words; outputting the modified textual representation of the audio data recording; and A system including:
7. the subunits include phonetic components of words; Reconstructing the unknown word comprises: consolidating the recognized subunits into a single unit bounded by an end time of a word immediately preceding the unknown word and a start time of a word immediately following the unknown word; looking up the unknown word using the single unit and a reverse phonetic dictionary; Including, The system of claim 6.
8. 7. The system of claim 6, wherein the subunits include fragments from which words can be constructed, each fragment including a word start symbol and a word end symbol, and wherein reconstructing the unknown words includes combining the recognized subunits into a single unit by concatenating the word end symbol of each subunit to the word start symbol of the immediately succeeding subunit.
9. 9. The system of claim 6, wherein identifying the unknown words in the text representation comprises determining that the word recognition model has output sub-word units for a segment of the speech data recording.
10. The system of any one of claims 6 to 9, wherein the operations further comprise adjusting interpolation weights of the word recognition model to adjust a preference for generating the text representation using whole words or subwords.
11. A computer readable storage medium having stored thereon a computer program for recovering undefined words in a transcript of an audio data recording, the computer readable storage medium having stored thereon a computer program comprising software code portions for performing the method of any one of claims 1 to 5.
12. A computer program loadable into the internal memory of a digital computer, the computer program comprising software code portions for performing the method according to any one of claims 1 to 5, when said program is run on a computer.
Citation Information
Patent Citations
Advancing word-based speech recognition processing
US20190279614A1
