Method and system for unsupervised discovery of unigrams in speech recognition systems
The unsupervised method enhances speech recognition systems by automatically discovering and integrating new words into the language model, addressing the limitation of recognizing only predefined vocabularies and improving transcription accuracy.
Patent Information
- Application Number
- JP2024525194
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-08
- Filing Date
- 2022-11-08
- Publication Date
- 2024-11-13
AI Technical Summary
Current speech recognition systems, such as LVCSR, are unable to recognize words outside their predefined vocabulary, including domain-specific terms like drug names and medical conditions, due to their reliance on supervised learning mechanisms that require extensive human intervention for updating domain-specific language models.
An unsupervised method for discovering unigrams in speech recognition systems using acoustic machine learning and a greedy decoder to predict initial unigrams, followed by filtering and updating the language model with candidate unigrams, and refining the acoustic model using beam decoding and n-gram extraction.
Automatically expands the vocabulary of speech recognition systems to include new words without manual intervention, improving transcription accuracy and adaptability to domain-specific terminology.
Smart Images

Figure 2024541957000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates generally to the field of speech recognition, and more particularly to unsupervised discovery of unigrams in speech recognition systems.
[0002] Claiming priority This application claims the benefit of U.S. patent application Ser. No. 17 / 520,816, entitled “METHOD AND SYSTEM FOR UNSUPERVISED DISCOVERY OF UNIGRAMS IN SPEECH RECOGNITION SYSTEMS,” filed with the U.S. Patent and Trademark Office on November 8, 2021. [Background technology]
[0003] In the field of speech analysis and recognition, Large Vocabulary Continuous Speech Recognition (LVCSR) systems are used to recognize spoken text and then generate a transcript of that spoken text by applying a vocabulary dictionary or language model of potentially 50,000-100,000 words and phrases.
[0004] Thus, currently available speech recognition systems such as LVCSR are, by design, unable to recognize out-of-vocabulary words. For example, a pharmaceutical company's call center LVCSR system would be unable to recognize drug names and terms describing medical conditions if these terms are not present in the LVCSR language model.
[0005] This point is further exacerbated by the fact that domain-specific terms that may normally be excluded from the global domain-independent language model may also have high importance in their specific field or domain. With respect to the pharmaceutical example, drug names and domain words such as "insulin", "antihistamine", "meningitis", "abdominal pain", "antibiotics" etc. may be highly relevant in the specific field of medicine and pharmacy, but nevertheless may not be included in the language model of the speech analysis system and therefore may not be recognized from the speech.
[0006] Currently available speech analysis systems such as LVCSR may maintain a supervised learning mechanism that allows for the expansion of the vocabulary "on-site" by manually adding phrases to the n-gram based language model in a continuous sequence to update and validate the domain-specific language model. It can be appreciated that this approach requires extensive human intervention. Summary of the Invention
[0007] Therefore, there is a need for a method and system for automatic, sequential, and unsupervised discovery of unigrams in a speech recognition system.
[0008] Embodiments of the present invention may include a method for automatically discovering unigrams within audio data elements by at least one processor.
[0009] An embodiment of the method includes receiving a language model, which may include a plurality of n-grams, where each n-gram may include one or more unigrams; applying an acoustic machine-learning (ML) model to the one or more first speech data elements to obtain a character distribution function; applying a greedy decoder to the character distribution function to predict an initial corpus of unigrams; filtering out one or more unigrams of the initial corpus to obtain a corpus of candidate unigrams not included in the language model; analyzing the one or more first speech data elements to extract at least one n-gram including the candidate unigram; and updating the language model to include the extracted at least one n-gram.
[0010] According to some embodiments, the at least one processor may apply a beam decoder to the second audio data element to generate at least one transcription of the second audio data element based on an updated language model (e.g., during the inference stage).
[0011] According to some embodiments, the character distribution function may represent the likelihood of occurrence of one or more linguistic characters in one or more first audio data elements.
[0012] According to some embodiments, the at least one processor may retrain the acoustic ML model based on the at least one second audio data element using the extracted at least one n-gram as training data.
[0013] According to some embodiments, at least one processor may filter out one or more unigrams by (a) calculating, for one or more candidate unigrams, a misspelling probability that represents the likelihood that an associated unigram is a misspelled version of a unigram already included in the language model, and (b) filtering out candidate unigrams that correspond to misspelling probabilities that exceed a predetermined threshold.
[0014] According to some embodiments, at least one processor may calculate the spelling error probability by calculating a Levenshtein distance value between the candidate unigram and at least one second unigram already included in the language model, calculating a frequency score representing an occurrence rate between the candidate unigram and the at least one second unigram in the one or more first speech data elements, and calculating the spelling error probability based on the Levenshtein distance value and the frequency score.
[0015] According to some embodiments, the greedy decoder may be adapted to emit a respective confidence level for each unigram of the initial corpus, in such an embodiment, the at least one processor may be configured to further calculate the misspelling probability by calculating a confidence score representing an average of the confidence levels for one or more occurrences of the candidate unigram in the one or more first speech data elements, and calculating the misspelling probability further based on the confidence score.
[0016] According to some embodiments, the language model may further include a definition of one or more language syntax rules. In such embodiments, the at least one processor may be configured to calculate the Levenshtein distance by calculating a number of single-character edits between the candidate unigram and the second unigram, and calculating a Levenshtein distance value based on the one or more language syntax rules and the number of single-character edits.
[0017] According to some embodiments, the at least one processor may calculate a missing space probability for one or more candidate unigrams. The missing space probability may represent a likelihood that the candidate unigram may be a concatenation of two unigrams already included in the language model. Additionally, the at least one processor may filter out candidate unigrams that correspond to a missing space probability that exceeds a predefined threshold.
[0018] According to some embodiments, at least one processor may calculate, for at least one first candidate unigram, a first unigram embedding vector based on the corpus of candidate unigrams, calculate a second unigram embedding vector for at least one second candidate unigram based on the corpus of candidate unigrams, calculate a similarity score based on the first unigram embedding vector and the second unigram embedding vector, and calculate a spelling error probability further based on the similarity score.
[0019] According to some embodiments, at least one processor may receive a document corpus including a plurality of documents, each document associated with a particular subject area, and each document including a plurality of document unigrams.
[0020] The at least one processor may calculate, for one or more candidate unigrams, an intra-domain score based on the multiple document unigrams, the intra-domain score representing a likelihood that the candidate unigram may be associated with at least one particular domain.
[0021] Additionally, for one or more candidate unigrams, the at least one processor may compile a context list that may include a subset of document unigrams. The subset of document unigrams (a) have an intradomain score above a predefined threshold and (b) may appear in one or more first speech data elements. For each document, the at least one processor may obtain an intersection group that may include the document and unigrams that appear in the context list. For each document, the at least one processor may calculate a precision score that represents the relevance of the candidate unigram to the document based on the intradomain scores of the document unigrams in the intersection group. The at least one processor may then filter out or omit candidate unigrams that correspond to a maximum precision score below the predefined threshold.
[0022] According to some embodiments, the greedy decoder may be adapted to emit a respective confidence level for each unigram of the initial corpus. In such an embodiment, at least one processor may analyze the speech data elements of one or more first speech data elements for one or more candidate unigrams by locating within the speech data elements n-grams of adjacent unigrams that may include the candidate unigram, and if (a) the unigram of the n-gram corresponds to a confidence level exceeding a predetermined value and (b) the n-gram includes more than a predetermined threshold number of unigrams, the language model may be updated to include the extracted at least one n-gram. Otherwise, the candidate unigram may be filtered out from the corpus of candidate unigrams.
[0023] An embodiment of the present invention may include a system for automatically discovering unigrams in audio data elements. An embodiment of the system may include a non-transitory memory device in which a module of instruction code may be stored, and at least one processor associated with the memory device and configured to execute the module of instruction code.
[0024] During execution of the modules of instruction code, the at least one processor may be configured to receive a language model, which may include a plurality of n-grams, each of which may include one or more unigrams; apply an acoustic ML model to the one or more first speech data elements to obtain a character distribution function; apply a greedy decoder to the character distribution function to predict an initial corpus of unigrams; filter out one or more unigrams of the initial corpus to obtain a corpus of candidate unigrams not included in the language model; analyze the one or more first speech data elements to extract at least one n-gram, which may include a candidate unigram; and update the language model to include the extracted at least one n-gram.
[0025] Additionally, embodiments of the system may include a beam decoder configured to receive the at least one second audio data element and generate a transcription of the at least one second audio data element based on the updated language model. [Brief description of the drawings]
[0026] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of this specification. The invention, however, together with its objects, features, and advantages, both as to organization and method of operation, may best be understood by reference to the following detailed description when read in conjunction with the accompanying drawings. [Figure 1] 1 is a block diagram illustrating a computing device that may be included in a system for speech recognition according to some embodiments of the present invention. [Diagram 2] 1 is a block diagram illustrating an overview of a system for speech recognition according to some embodiments of the present invention. [Diagram 3] 1 is a block diagram illustrating data flow in a system for speech recognition according to some embodiments of the present invention. [Figure 4] 1 is a block diagram illustrating another view of a system for speech recognition according to some embodiments of the present invention. [Diagram 5] 1 is a block diagram illustrating a unigram extraction module that may be included in a system for speech recognition in text according to some embodiments of the present invention. [Figure 6] 1 is a heat map illustrating the probability of substitution and deletion of letters of the English alphabet in a system for speech recognition according to some embodiments of the present invention. [Figure 7] FIG. 2 is a block diagram illustrating another example of a unigram extraction module that may be included in a system for speech recognition according to some embodiments of the present invention. [Figure 8]1 is a flow diagram illustrating a method for automatically discovering unigrams in speech data elements by a system for speech recognition according to some embodiments of the present invention.
[0027] It will be understood that for simplicity and clarity of illustration, elements illustrated in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0028] Those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The foregoing embodiments are therefore to be considered in all respects illustrative rather than limiting of the invention described herein. The scope of the present invention is therefore indicated by the appended claims rather than by the foregoing description, and all changes that come within the meaning and range of equivalence of the claims are therefore intended to be embraced therein.
[0029] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For clarity, descriptions of the same or similar features or elements may not be repeated.
[0030] For example, although embodiments of the invention are not limited in this respect, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "establishing," "analyzing," "checking," etc. may refer to operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulates and / or transforms data represented as physical (e.g., electronic) quantities in the computer's registers and / or memory into other data similarly represented as physical quantities in the computer's registers and / or memory, or other information non-transitory storage media that may store instructions for performing an operation(s) and / or process(es).
[0031] Although embodiments of the invention are not limited in this respect, the terms "plurality" and "a plurality" as used herein may include, for example, "multiple" or "two or more." The terms "plurality" or "a plurality" may be used throughout this specification to describe two or more components, devices, elements, units, parameters, etc. The term "set" as used herein may include one or more items.
[0032] Unless expressly stated, the method embodiments described herein are not constrained to a particular order or sequence. Additionally, some of the method embodiments described or elements thereof may occur or be performed simultaneously, at the same time, or in parallel.
[0033] Table 1 below contains a glossary of terms used herein.
[0034] [Table 1]
[0035] Reference is now made to FIG. 1, which is a block diagram illustrating a computing device that may be included in a system for speech recognition, according to some embodiments.
[0036] Computing device 1 may include a processor or controller 2, which may be, for example, a central processing unit (CPU) processor, chip, or any suitable computing or computational device, an operating system 3, memory 4, executable code 5, a storage system 6, input devices 7, and output devices 8. Processor 2 (or one or more controllers or processors, possibly across multiple units or devices) may be configured to perform the methods described herein and / or to perform or operate as various modules, units, etc. More than one computing device 1 may be included in a system according to embodiments of the invention, and one or more computing devices 1 may function as components of a system according to embodiments of the invention.
[0037] Operating system 3 may be or include any code segment (e.g., similar to executable code 5 described herein) designed and / or configured to perform tasks that involve coordinating, scheduling, arbitrating, supervising, controlling, or managing the operation of computing device 1, such as scheduling the execution of software programs or tasks, or enabling software programs or other modules or units to communicate. Operating system 3 may be a commercial operating system. It should be noted that operating system 3 may be any component, for example, in some embodiments a system may include a computing device that does not require or include an operating system 3.
[0038] The memory 4 may be or include, for example, a Random-Access Memory (RAM), a read only memory (ROM), a Dynamic RAM (DRAM), a Synchronous DRAM (SD-RAM), a double data rate (DDR) memory chip, a flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short-term storage unit, a long-term storage unit, or other suitable memory or storage unit. The memory 4 may be or include a number of possible different memory units. The memory 4 may be a computer or processor non-transitory readable medium, or a computer non-transitory storage medium, for example, a RAM. In one embodiment, the non-transitory storage medium, such as the memory 4, a hard disk drive, or another storage device, may store instructions or code that, when executed by the processor, may cause the processor to perform the methods described herein.
[0039] Executable code 5 may be any executable code, such as an application, a program, a process, a task, or a script. Executable code 5 may be executed by processor or controller 2, perhaps under the control of operating system 3. For example, executable code 5 may be an application that may perform speech recognition as described further herein. For clarity, although a single item of executable code 5 is shown in FIG. 1, systems according to some embodiments of the invention may include multiple executable code segments similar to executable code 5 that may be loaded into memory 4 and cause processor 2 to perform the methods described herein.
[0040] Storage system 6 may be or may include, for example, a flash memory as known in the art, a memory within or embedded in a microcontroller or chip as known in the art, a hard disk drive, a CD-Recordable (CD-R) drive, a Blu-ray disk (BD), a universal serial bus (USB) device, or other suitable removable and / or fixed storage unit. Data relating to one or more audio data elements may be stored in storage system 6, loaded from storage system 6 into memory 4, and processed by processor or controller 2. In some embodiments, some of the components shown in FIG. 1 may be omitted. For example, memory 4 may be a non-volatile memory having the storage capacity of storage system 6. Thus, although shown as a separate component, storage system 6 may be embedded in or included in memory 4.
[0041] Input device(s) 7 may be or include any suitable input device, component, or system, such as a detachable keyboard or keypad, a mouse, etc. Output device(s) 8 may include one or more (possibly detachable) displays or monitors, speakers, and / or any other suitable output device. Any applicable input / output (I / O) devices may be connected to computing device 1, as indicated by blocks 7 and 8. For example, a wired or wireless network interface card (NIC), a universal serial bus (USB) device, or an external hard drive may be included in input device(s) 7 and / or output device(s) 8. It will be appreciated that any suitable number of input devices 7 and output devices 8 may be operatively connected to computing device 1, as indicated by blocks 7 and 8.
[0042] Systems according to some embodiments of the present invention may include components such as, but not limited to, multiple central processing units (CPUs) or any other suitable general-purpose or specific processors or controllers (e.g., similar to element 2), multiple input units, multiple output units, multiple memory units, and multiple storage units.
[0043] Recently, a new approach to speech recognition, commonly referred to as "end-to-end recognition", has opened up another option for detecting new words. Unlike traditional speech recognition methods that rely on word-level graphs to perform recognition, where every word is built from a sequence of phonemes, end-to-end recognition has switched completely to the character level. This means that instead of an output that is a sequence of words or phonemes (or a graph, a "lattice"), end-to-end speech recognition outputs a character distribution function.
[0044] Reference is now made to Figure 2, a block diagram illustrating an overview of a system 100 for speech recognition, in accordance with some embodiments of the present invention. As detailed herein, the system 100 may be configured to automatically discover unigrams (e.g., words) within one or more speech data elements to improve a given language model 160A, and then utilize the improved language model 160A in conjunction with a beam decoder to perform speech recognition on one or more new speech data element samples.
[0045] As depicted in FIG. 2, system 100 may implement end-to-end speech recognition while leveraging synergies between three separate functions (marked by dashed lines): acoustic analysis, greedy decoding, and beam decoding, as detailed herein.
[0046] According to some embodiments of the invention, system 100 may be implemented as software modules, hardware modules, or any combination thereof. For example, system 100 may be or may include a computing device, such as element 1 of FIG. 1, adapted to execute one or more modules of executable code (e.g., element 5 of FIG. 1) to perform speech recognition, as described further herein.
[0047] As shown in Figure 2, arrows may represent the flow of one or more data elements to and / or from system 100 and / or between modules or elements of system 100. Some arrows have been omitted from Figure 2 for clarity.
[0048] As shown in FIG. 2, the system 100 may receive an initial version of the language model 160A (e.g., from the input device 7 of FIG. 1). According to some embodiments, the language model 160A may include multiple n-grams, each of which may include one or more unigrams. For example, as is known in the art of natural language processing (NLP), the n-gram-based language model 160A may include multiple n-grams, each of which may represent a set or combination of unigrams (e.g., words). Each n-gram may have a probability associated with it that represents the probability or frequency of occurrence of each combination of unigrams in a corpus of documents. For example, the n-gram probability of the n-gram "I love you" may be higher than the n-gram probability of the n-gram "I love pancakes," representing a higher likelihood that the unigram "you" will appear in the speech data element 20 following the n-gram "I love."
[0049] According to some embodiments, system 100 may receive (e.g., via input device 7 of FIG. 1 ) one or more acoustic or speech data elements 20. The one or more acoustic or speech data elements 20 may be, for example, audio files or data streams that may include or represent human speech in a natural language (e.g., English, French, etc.). As detailed herein, system 100 may be configured to perform unsupervised discovery of unigrams in the one or more speech data elements 20 and thereafter generate transcription data element 100A representing an audio transcription of the one or more received acoustic or speech data elements 20.
[0050] According to some embodiments, the system 100 may include a feature extraction module 112 adapted to extract one or more acoustic features 112A from the speech 20.
[0051] For example, the feature extraction module 112 may be configured to extract one or more acoustic features 112A, such as Mel-frequency cepstral coefficients (MFCCs), from the speech 20. As is known in the art, MFCC coefficients may be derived from the cepstrum of acoustic data elements, such as a stream of acoustic signals. The cepstrum may be calculated as the inverse Fourier transform (IFT) of the logarithm of the frequency spectrum of the acoustic signal.
[0052] 2, the feature extraction module 112 may feed the acoustic features 112A (e.g., MFCCs) to a machine learning (ML)-based acoustic model 110. The system 100 may apply the acoustic model 110 to one or more speech data elements 20 and / or the corresponding acoustic features 112A to generate or predict (as is commonly referred to in the art) a character distribution function 110A. The character distribution function 110A may represent a probability distribution of linguistic characters (e.g., [a-z]) across a given alphabet based on the input acoustic features 112A. In other words, the character distribution function 110A may represent the likelihood of occurrence of one or more linguistic characters in one or more speech data elements 20.
[0053] As is known in the art, the acoustic model 110 may predict a character distribution function 110A for a particular time window (e.g., 25 milliseconds (ms)) of a data element of the input speech 20. In other words, for each time window of the input speech 20, the ML-based acoustic model 110 may generate a character distribution function 110A for language characters (and / or non-speech or "blank" portions within the speech 20) that represents the probability that a particular character or characters was spoken or pronounced during that time window.
[0054] According to some embodiments, the acoustic model 110 may include an ML-based model, such as a convolutional neural-network (CNN) model, a deep convolutional neural network (DNN) model, a recurrent neural network (RNN) model, an attention-based neural network model, or any other suitable ML model for predicting the character distribution function 110A as known in the art.
[0055] According to some embodiments, the system 100 may include a greedy decoder module 120 adapted to receive a character distribution function 110A from the acoustic model 110 associated with a particular time window of the underlying speech data element 20.
[0056] As known in the art, the greedy decoder 120 may be configured to first remove repetitions of characters and / or occurrences of non-speech or "blank" portions within a given time window. The greedy decoder 120 may then determine the most likely combinations or sequences of remaining characters within a given time window and / or among multiple consecutive time windows to generate one or more unigrams 120A or words based on the character distribution function 110A. For example, the greedy decoder 120 may generate one or more unigrams 120A by selecting a sequence of characters that corresponds to a maximum probability of occurrence as reflected by the character distribution function 110A.
[0057] Additionally, the greedy decoder 120 may associate or attribute unigram metadata 120A' to each generated unigram 120A. For example, the unigram metadata 120A' may include a time frame corresponding to each generated unigram or word 120A, defining the start and end times of the sequence of characters that includes the associated unigram 120A.
[0058] In another example, the greedy decoder 120 may calculate, for one or more (e.g., each) generated unigram, a unigram metadata 120A' element that represents a confidence level or confidence score (e.g., a numeric value in the range [0,1]) for the occurrence of the unigram 120A in the underlying audio data element 20. The confidence score may be calculated, for example, as a function of the probability of the characters included in the unigram (e.g., as reflected in the character distribution function 110A), and may be normalized, for example, by the length (e.g., number of characters) of the associated unigram. The greedy decoder 120 may then assign or attribute the confidence score as metadata 120A' for the associated unigram 120A.
[0059] According to some embodiments, the system 100 may successively apply the greedy decoder 120 to one or more instances of the character distribution function 110A to generate or predict an initial corpus of unigrams 120A (and corresponding metadata 120A'). The term "successively" may be used in this context to indicate that the greedy decoder 120 may be applied to multiple instances of the character distribution function 110A resulting from multiple respective time periods within a single audio data element 20. Additionally or alternatively, the greedy decoder 120 may be applied to multiple instances of the character distribution function 110A resulting from multiple audio data elements 20.
[0060] It may be understood by those skilled in the art that a greedy decoder such as greedy decoder 120 may be capable of recognizing any character sequence, but may tend to output misspelled words (e.g., "termometer" instead of "thermometer"). In other words, the initial corpus of unigrams 120A may include misspelled words. Conversely, a beam decoder such as beam decoder 170 may not generate or emit transcriptions that include misspelled words, but may be limited by a finite, but large, vocabulary or language model 160A.
[0061] Reference is now made to Figure 3, which is a block diagram illustrating the flow of data within a system 100 for speech recognition, according to some embodiments of the present invention. It can be understood that the system 100 of Figure 3 may be the same system 100 illustrated in Figure 2, with some of the modules and elements omitted for clarity.
[0062] According to some embodiments, the greedy decoder 120 may work with a filter module 121 configured to apply a filter to an initial corpus of unigrams 120A to filter out or exclude unigrams (e.g., words) already included in the language model 160A. This filtering may produce a corpus of candidate unigrams, denoted herein as unigrams 121A, and corresponding metadata 121A′.
[0063] As described in more detail herein (e.g., in relation to FIG. 4), the system 100 may apply a set of filters to the candidate unigrams 121A to generate one or more filtered unigrams, denoted herein as filtered unigrams 130A, 140A, and corresponding metadata (130A', 140A') describing the filtered unigrams 130A, 140A. As described in more detail herein (e.g., in relation to FIG. 4), the system 100 may analyze the acoustic speech data elements 20 and / or the filtered unigrams 130A / 140A to generate or extract at least one n-gram 150A, which may include one or more filtered unigrams 130A / 140A. The system 100 may then update or improve the language model 160A to include the at least one n-gram 150A.
[0064] 3, the system 100 may include or cooperate with a beam decoder 170. The system 100 may enable the beam decoder 170 to utilize the improved language model 160A to generate an error-free transcription 100A of the acoustic speech data element 20 based on the improved language model 160A. In other words, the system 100 may apply the beam decoder 170 to at least one new speech data element 20 to generate at least one transcription 100A of the new speech data element based on the updated language model 160A.
[0065] Additionally, as described in more detail herein, system 100 may utilize extracted n-grams 150A along with corresponding acoustic segments of speech data elements 20 (e.g., marked by the start and end times of the n-grams within speech data elements 20) as feedback for acoustic model 110 to fine-tune or retrain acoustic model 110. Thus, system 100 may automatically generate a more accurate prediction of character distribution function 110A based on the automatic extraction of n-grams 150A.
[0066] Embodiments of the present invention may have practical applications in performing speech recognition and transcription tasks, and may include several improvements over currently available methods and systems for speech recognition.
[0067] For example, by synergistically combining the advantages of a greedy decoder and a beam decoder, embodiments may automatically (e.g., without the need for manual supervision) identify new unigrams or words that are not present in the language model 160A, improve the language model or vocabulary 160A to include these new unigrams or words, and generate error-free transcriptions of speech data elements that include the newly identified unigrams.
[0068] Additionally, embodiments of the present invention may take advantage of this advantage of automatic unigram identification to continuously (e.g., repeatedly through time) retrain or refine an underlying acoustic model, such as acoustic model 110. It may be appreciated that such an acoustic model may be the basis of any speech recognition paradigm. Thus, refining or fine-tuning the acoustic model may result in generating a text transcription 100A that may be agnostic to, for example, the speaker's locale or accent.
[0069] 4 is a block diagram illustrating a system 100 for automatic topic detection in text, according to some embodiments of the present invention. It can be understood that the system 100 of FIG. 4 can be the same system 100 as that illustrated in FIG. 2 and / or FIG. 3.
[0070] 4, the system 100 may include two complementary unigram extraction modules, shown as unigram extraction modules 130 and 140. The unigram extraction modules 130 / 140 may receive a corpus of candidate unigrams 121A from the greedy decoder 120 that lacks unigrams already included in the language model or vocabulary 160A. At least one (e.g., each) unigram extraction module 130 / 140 may be configured to classify the received candidate unigrams 121A as either a misspelled word or a new word that is likely correct in the appropriate context of the speech data element 20.
[0071] According to some embodiments, the first unigram extraction module (e.g., 140) may utilize an external text corpus, such as Wikipedia summaries, to filter out words that are likely to be misspelled, and the second unigram extraction module (e.g., 130) may utilize a set of misspelling filters without the need for such an external text corpus. It may be understood that the system 100 may use any one of these features separately, for example, depending on the availability of an external text corpus. Additionally or alternatively, the system 100 may synergistically combine the features of the unigram extraction modules 130 and 140 to generate new unigrams (e.g., words) that are likely to be correctly spelled to improve the n-gram based language model 160A.
[0072] Reference is now also made to FIG. 5, which is a block diagram illustrating an example of a unigram extraction module 130 that may be included in a system 100 for speech recognition, according to some embodiments of the present invention. As shown in the example of FIG. 5, the unigram extraction module 130 may include one or more (e.g., a set or cascade) of several (e.g., four) different filters, shown herein as an initial candidate generator 131, a Levenshtein distance-based filter 133, a missing space filter 135, and a candidate pair filter 137. It may be understood that the order of filtering (e.g., the order of the filter sets) may vary among embodiments of the present invention. Other filter types and combinations are possible. As detailed herein, one or more filters (e.g., 131, 133, 135, and / or 137) may filter a group or list of candidate unigrams 121A to generate a list or group of filtered unigrams 130A and corresponding metadata 130A′.
[0073] According to some embodiments, the initial candidate generator 131 may be configured to filter out candidate unigrams 121A that (a) are already included in the language model or vocabulary 160A or (b) have confidence score metadata 121A' below a predefined confidence threshold (e.g., 0.95).
[0074] Additionally or alternatively, the initial candidate generator 131 may be configured to filter out unigrams 121A below a predefined minimum count (e.g., 5). At this stage, it may be appreciated that the system 100 may typically include hundreds or thousands of unique candidate unigrams. Some of these unique candidate unigrams may be correctly spelled unigrams (e.g., words), while others may be misspelled.
[0075] As described in detail herein, the unigram extraction module 130 (and / or 140) may be configured to calculate, for one or more (e.g., each) candidate unigram 121A, a misspelling probability that represents the likelihood that the associated candidate unigram 121A is a misspelled version of a unigram already included in the language model 160A. The unigram extraction module 130 (and / or 140) may filter out candidate unigrams 121A that correspond to a misspelling probability that exceeds a predefined threshold. In other words, the unigram extraction module 130 (and / or 140) may continuously (e.g., repeatedly over time) filter out candidate unigrams 121A such that more and more of the misspelled or incorrectly spelled words relative to correctly spelled words may be removed.
[0076] Table 2 Table 2 below details examples of various types of misspelled unigrams or words 121A that may be generated by the greedy decoder and that may be addressed or corrected by the system 100 for speech recognition, according to some embodiments of the present invention. The examples provided in Table 2 represent experimentally encountered examples of misspelled, frequently occurring, high confidence unigrams generated by the greedy decoder 120.
[0077] [Table 2]
[0078] As is known in the art, the Levenshtein distance is a metric that can be used to measure the difference between two sequences. For example, the Levenshtein distance between two words is the minimum number of single-character edits (e.g., insertions, deletions, or substitutions) required to change one word into the other.
[0079] According to some embodiments, the Levenshtein-based filter 133 may be configured to process (e.g., remove) misspelled candidate unigrams 121A that manifest a change (e.g., a substitution, insertion, and / or deletion) in at least one character of a correctly spelled word.
[0080] For the examples presented in Table 2, examples 1-7 represent misspelled unigrams 121A (e.g., "Forard") having spelling errors that occur due to character substitutions, insertions, and / or deletions (e.g., with respect to "Forward"). In other words, the Levenshtein-based filter 133 may be configured to filter out at least one candidate unigram 121A that is sufficiently close (e.g., in terms of Levenshtein distance) to an in-vocabulary word.
[0081] It can be appreciated that Levenshtein distance filtering as known in the art may filter out unigrams 121A based solely on the Levenshtein distance metric between unigrams and may prove to be too aggressive in the sense that it may unnecessarily exclude legitimate (e.g., correctly spelled) words. For example, if multiple correctly spelled in-vocabulary variants exist for a word, currently available Levenshtein-based filters may match a legitimate candidate unigram 121A to one variant of the word and thus erroneously disqualify and filter out the candidate unigram 121A.
[0082] According to some embodiments, the Levenshtein-based filter 133 may implement one or more additional tests to avoid such errors and thus provide an improvement over Levenshtein distance filters known in the art.
[0083] For example, the Levenshtein-based filter 133 may calculate a Levenshtein distance value 133A between the candidate unigram 121A and at least one in-vocabulary unigram already included in the language model 160A. The Levenshtein-based filter 133 may then calculate a frequency score 133B representing a rate of occurrence between the candidate unigram 121A and the at least one in-vocabulary unigram in one or more speech data elements 20. The Levenshtein-based filter 133 may then calculate a spelling error probability 133E based on the Levenshtein distance value 133A and the frequency score value 133B.
[0084] For example, the Levenshtein-based filter 133 may calculate a Levenshtein distance value between the candidate unigrams 121A and the unigrams already included in the language model 160A. The Levenshtein-based filter 133 may identify one or more such pairs as similar pairs according to the distance metric value (e.g., having a Levenshtein distance value below a predefined threshold). The Levenshtein-based filter 133 may preserve the candidate unigrams 121A that have higher frequency score values 133B (e.g., occur more frequently in the speech data elements 20) compared to similar pair unigrams of the in-vocabulary unigrams 160A.
[0085] For example, if an out-of-vocabulary (OOV) candidate unigram 121A such as "pressurizing" appears 15 times in the corpus of input speech data elements 20, and an in-vocabulary (e.g., included in language model 160A) unigram such as "pressuring" appears 2 times in the corpus of input speech data elements 20, the Levenshtein-based filter 133 may retain the "pressurizing" candidate unigram 121A. In another example, if an OOV candidate unigram 121A such as "insect" appears 17 times in the corpus of input speech data elements 20, and an in-vocabulary unigram such as "inject" appears 3580 times, the Levenshtein-based filter 133 may filter out "insect" from the list of candidate unigrams 121A.
[0086] As described in more detail herein (e.g., in relation to FIG. 2), the greedy decoder 120 may be configured to emit or compute, for each predicted unigram of the initial corpus of unigrams 120A, a respective unigram metadata 120A′ element that represents a confidence level of the predicted unigram. According to some embodiments, the Levenshtein-based filter 133 may further compute a misspelling probability 133E based on the confidence level (e.g., in addition to the Levenshtein distance value 133A and the frequency score value 133B).
[0087] For example, the Levenshtein-based filter 133 may calculate an average confidence score 133D that represents an average of the confidence levels 120A' for one or more occurrences of the candidate unigrams 121A in the speech data element corpus 20. The Levenshtein-based filter 133 may calculate a weighted sum of the frequency score values 133B average confidence scores 133D. The Levenshtein-based filter 133 may then retain the candidate unigrams 121A that have a higher weighted sum of the frequency score values 133B average confidence scores 133D compared to their in-vocabulary unigrams 160A similar paired unigrams.
[0088] Additionally or alternatively, the Levenshtein-based filter 133 may sort the candidate unigrams 121A based on a predefined set of specialized language-specific rules.
[0089] For example, language model 160A may include definitions of one or more language-specific syntax rules, such as for English, which may define verb conjugations for time (e.g., adding "ed" to the end of a verb to indicate the past tense), plural vs. singular conjugations (e.g., adding "s" to indicate plural nouns), etc.
[0090] In such an embodiment, the Levenshtein-based filter 133 may calculate the number of single-character edits or changes between the candidate unigrams 121A and the in-vocabulary unigrams and calculate a Levenshtein distance value 133A based on one or more language syntax rules and the number of single-character edits.
[0091] With respect to the above usage example, the Levenshtein-based filter 133 may sort the candidate unigrams 121A based on time conjugation (e.g., past tense "evaluated" vs. present tense "evaluates") and / or plural vs. singular conjugation (e.g., "prescription" vs. "prescriptions"), etc. Other such rules for sorting the candidate unigrams 121A may also be implemented.
[0092] It can be appreciated that currently available Levenshtein distance filters known in the art may not distinguish between types of character variations.
[0093] For example, "m" substituted with "n" may be a frequent error due to the phonetic similarity of the two letters. In contrast, the substitution of "d" with "o" may occur less frequently. Thus, substituting "m" with "n" is more likely to represent a spelling error than substituting "d" with "o". In another example, the erroneous insertion of the vowel "i" in currently available systems of speech recognition may occur more frequently than the insertion of the consonant "t" and is therefore more likely to represent a spelling error.
[0094] According to some embodiments, the Levenshtein-based filter 133 may take into account the frequency of such changes (e.g., substitutions, deletions, and / or insertions) and thus provide an improvement over Levenshtein distance filters known in the art. For example, the unigram extraction module 130 may quantify character changes (e.g., substitutions, deletions, and / or insertions) by calculating change probabilities 133C based on character-level alignments of greedy decoder transcriptions versus human transcriptions, as detailed herein (e.g., in connection with FIG. 6).
[0095] Reference is now made to FIG. 6, which is an example heatmap illustrating the probability of changes 133C (e.g., substitutions and / or deletions) of letters of the English alphabet in a system for speech recognition according to some embodiments of the present invention.
[0096] The example heatmap in Figure 6 was obtained based on several hours of acoustic speech data elements and corresponding transcriptions. The change probability 133C for each character was calculated as the ratio between the frequency or number of change occurrences and the total number of occurrences of the character in the corpus of speech data elements 20.
[0097] For example, the letters "z" and "s" have high substitution probabilities, as do "q" and "c," as shown in the heatmap in Figure 6. In another example, the apostrophe (') character has a very large deletion probability.
[0098] According to some embodiments, the Levenshtein-based filter 133 may calculate a Levenshtein distance value 133A between the candidate unigram 121A and at least one in-vocabulary unigram already included in the language model 160A. For each character that contributes to the Levenshtein distance value 133A, the Levenshtein-based filter 133 may calculate a probability of an underlying change 133C. The Levenshtein-based filter 133 may calculate a misspelling probability 133E based on the Levenshtein distance value 133A and further based on the frequency score value 133B and / or the change probability 133C. For example, the misspelling probability 133E may be calculated as a weighted sum of the Levenshtein distance values 133A weighted by the change probability 133C. The Levenshtein-based filter 133 may then filter out one or more candidate unigrams 121A based on the misspelling probability 133E.
[0099] As shown in the examples of Table 2 (e.g., entries 8 and 9), the greedy decoder may generate a misspelled candidate unigram 121A due to the omission of a space or "white space" between two consecutive unigrams (e.g., words). For example, the greedy decoder 120 may omit the white space between the words "Thank" and "you" to generate a misspelled candidate unigram 121A such as "Thankyou."
[0100] According to some embodiments, the unigram extraction module 130 may include a missing space filter module 135 configured to omit or filter out such misspelled candidate unigrams 121A.
[0101] In other words, the missing space filter module 135 may calculate, for one or more candidate unigrams 121A, a missing space probability that represents the likelihood that the candidate unigram 121A is a concatenation of two unigrams already included in the language model 160A. The missing space filter module 135 may then filter out candidate unigrams 121A that correspond to missing space probabilities 135A that exceed a predefined threshold.
[0102] For example, the missing space filter module 135 may calculate (a) the frequency of occurrence of the candidate unigram 121A (e.g., "Thankyou") in the corpus of speech data elements 20, and (b) the frequency of occurrence of concatenated bigrams (e.g., consisting of consecutive unigrams such as "Thank" and "you") that generate the same unigram as the candidate unigram 121A (e.g., "Thankyou") after concatenation. The missing space filter module 135 may calculate a missing space probability 135A based on these calculated frequencies of occurrence, and may filter out candidate unigrams 121A corresponding to missing space probabilities that exceed a predefined threshold.
[0103] Additionally or alternatively, the missing space filter 135 may filter out from the list of candidate unigrams 121A all “concatenated” candidate unigrams 121A (e.g., “Thankyou”) that are less frequent than the corresponding combination of disjoint unigrams (e.g., “Thank” and “you”). In this example, the logic behind such filtering relies on the fact that “Thankyou” appears more frequently than “Thankyou” in the corpus of speech data elements 20, and therefore “Thankyou” is likely a misspelling and should be removed.
[0104] According to some embodiments, the unigram extraction module 130 may include a candidate pair filter module 137 configured to omit or filter out candidate unigrams 121A that do not have a paired unigram in the known vocabulary of the language model 160A but have a more frequent paired candidate unigram 121A in the list of candidate unigrams 121A.
[0105] According to some embodiments, the candidate pair filter module 137 may calculate a distance metric value (e.g., a Levenshtein distance value as detailed herein) between one or more (e.g., each) pairs of candidate unigrams 121A. The candidate pair filter module 137 may identify one or more such pairs as similar pairs according to the distance metric value, e.g., having a Levenshtein distance value below a predefined threshold. For each member (e.g., candidate unigram 121A) of each similar pair, the candidate pair filter module 137 may calculate a frequency of occurrence in the corpus of speech data elements 20 and then remove or filter out less frequent members.
[0106] For example, language model 160A may initially lack the unigram (e.g., word) "diabetes." Over time, the list of candidate unigrams 121A may evolve to include the correct unigram candidate 121A "diabetes" and the misspelled unigram candidate 121A "diabete." In this example, candidate pair filter module 137 may be configured to filter out or omit the misspelled candidate unigram 121A "diabete" from the list of candidate unigrams 121A and retain the correct candidate unigram 121A "diabetes" for further analysis, as described in more detail herein.
[0107] Reference is now also made to FIG. 7, which is a block diagram illustrating another example of a unigram extraction module 140 that may be included in the system 100 for speech recognition according to some embodiments of the present invention.
[0108] 7 may represent the flow of data between modules of the unigram extraction module 140 and / or the flow of data to or from the unigram extraction module 140. Some arrows have been omitted for clarity.
[0109] As shown in FIG. 7, the unigram extraction module 140 may receive one or more (e.g., a list) of candidate unigrams 121A from the greedy decoder 120 and may apply a filtering process to the list of candidate unigrams 121A to obtain one or more filtered unigrams 140A and corresponding metadata.
[0110] According to some embodiments, the unigram extraction module 140 may receive (e.g., from the input device 7 of FIG. 1 ) an external document corpus that may include a plurality of text documents 40. The term “external” may be used in this context to indicate that the documents 40 may not belong to the same subject matter domain as the audio data elements 20 and may not be the result of transcription by the system 100.
[0111] Each document 40 may include multiple document unigrams (e.g., words) 40A. For example, the document corpus may include multiple online documents 40 that are paper summaries, such as Wikipedia summaries, and the document unigrams 40A may be words included in the Wikipedia summaries. As described in more detail herein, the unigram extraction module 140 may utilize the corpus of text documents 40 to determine which candidate unigrams 121A generated by the greedy decoder 120 are likely to be erroneous and filter them out.
[0112] According to some embodiments, the unigram extraction module 140 may include an embedding model 142, such as a word2vec embedding model 142, as is known in the art.
[0113] The embedding model 142 may be trained based on the corpus of unigrams (e.g., unigrams 120A, 121A) generated by the greedy decoder 120 to compute or emit an embedding vector 142A. In other words, for each sample of a candidate unigram 121A, the embedding model 142 may generate an embedding vector 142A, which may be or may include a vector representation of the semantic meaning of the corresponding candidate unigram 121A.
[0114] As described in detail herein, the unigram extraction module 140 may utilize embedding vector similarity scores along with Levenshtein distance, intra-region scores, and unigram frequencies to reduce noise coming from the greedy decoder 120 and the corpus of documents 40.
[0115] A combination of such tools may be required because a document 40 (e.g., a Wikipedia summary) may contain unigrams 40A (e.g., words) that may be produced by the greedy decoder 120 but may nevertheless be inaccurate in the context of a transcription of an audio data element 20.
[0116] For example, speech data element 20 may contain the word "going," which may be mistranslated by greedy decoder 120 as "gong," which itself is a correctly spelled word that may appear within the corpus of external documents 40. From this example, it may be apparent that the combination of (a) a sufficiently small Levenshtein distance and (b) a sufficiently large embedding vector similarity value or intra-region score may indicate a high probability of a misspelled unigram.
[0117] In other words, as described in detail herein, embodiments of the present invention may calculate a misspelling probability based on, or as a function (e.g., a weighted sum) of, the Levenshtein distance value 133A, the embedding similarity score 142B, the intra-region score, and the frequency of occurrence.
[0118] It may be understood by one skilled in the art that the extraction of unigrams performed by the unigram extraction module 140 may be complementary to and may operate synergistically with the extraction of unigrams performed by the unigram extraction module 130.
[0119] For example, the unigram extraction module 130 may not rely on a sufficiently large corpus of external documents 40. On the other hand, the unigram extraction module 140 may allow the initial confidence level 120A′ of the candidate unigrams (relative to that used for the unigram extraction module 130) to be reduced, thus allowing more candidates to be considered and broadening the final set of found words.
[0120] According to some embodiments, the unigram extraction module 140 may include a preprocessing module 141 configured to process or align the syntax of the document unigrams 40A and / or the candidate unigrams 121A.
[0121] For example, the preprocessing module 141 may prepare the external document 40 (eg, a Wikipedia summary) to be lowercased, without punctuation or special characters.
[0122] In another example, the preprocessing module 141 may find collocations in the greedy decoder result corpus of candidate unigrams 121A (e.g., "troubleshoot") and prepare a clone 121A' of the corpus in which all found collocated unigrams are "glued" with underlines (e.g., troubleshoot). This may be done by analyzing the frequency of co-occurrences versus single occurrences in the corpus of speech data elements 20 and then determining whether collocations are found based on a predefined threshold.
[0123] According to some embodiments, an embedding model (eg, a word2vec model) 142 may be trained based on the corpus clone unigrams 121A′ to generate embedding vectors 142A.
[0124] According to some embodiments, system 100 may distinguish between two types of unigrams, which may be referred to herein as intra-domain unigrams and background unigrams. The term "intra-domain" may be used in this context to denote unigrams that are within the particular subject domain in which system 100 is configured to operate. The term "background" may be used in this context to denote unigrams that may extend beyond the particular subject domain in which system 100 is configured to operate.
[0125] For example, the system 100 may be deployed to perform speech recognition, such as in a pharmaceutical company's call center. In such an application, the speech data elements 20 may include discussions related to the pharmaceutical and medical subject domain. In this regard, the within-domain unigrams may include words taken from the pharmaceutical and medical subject domain, such as "insulin," "heartrate," "leukocytes," and "emphysema."
[0126] For the same example, common unigrams such as "hello," "thanks," "no," "not," "what," and "today" are not unique or specific to any subject domain and therefore may be referred to as background domain or out-of-domain words, etc. It may be appreciated that the abundance of occurrences of such words across multiple subject domains is what makes these words "noisy" in the sense that they may add extraneous data to the processing of the speech data elements 20. Additionally, words taken from the banking and insurance subject domain, such as "mortgage" and "credit," may be considered background domain unigrams since they are not within-domain unigrams for the pharmacy and medicine subject domain.
[0127] According to some embodiments, the unigram extraction module 140 may include a background corpus generation module 143 (or background module 143 for short). The background module 143 may be configured to receive a plurality of background text corpora 20C'. For example, the background text corpus 20C' may include a plurality of text data elements obtained via transcription of one or more (e.g., a plurality) background speech data elements 20C. The term "background" may be used in this context to indicate that the speech data element 20C and / or the subsequent text corpus 20C' may not be related to the subject domain of the speech data element 20.
[0128] According to some embodiments, background module 143 may filter out low frequency unigrams from received text corpus 20C' to generate a corpus of background unigrams 143A. The motivation for such filtering may be filtering (e.g., omission) of misspelled unigrams in received text corpus 20C'.
[0129] According to some embodiments, the unigram extraction module 140 may include an intra-region score calculator adapted to calculate an intra-region score 144A for each candidate unigram 121A in the greedy decoded corpus 120A.
[0130] For example, the unigram extraction module 140 may also calculate a second frequency of occurrence of the candidate unigram 121A in the greedy decoder corpus of unigrams 120A. The unigram extraction module 140 may normalize (e.g., divide) the second frequency of occurrence by the total number of documents or speech data elements 20. This frequency of occurrence may be referred to herein as the foreground normalized document frequency 144B. The unigram extraction module 140 may calculate a first frequency of occurrence of the candidate unigram 121A in the corpus of background unigrams 143A. The unigram extraction module 140 may normalize (e.g., divide) the first frequency of occurrence by the total number of documents in the received text corpus 20C'. This frequency of occurrence may be referred to herein as the background normalized document frequency 144C. The unigram extraction module 140 may then divide the foreground normalized document frequency 144B by the background normalized document frequency 144C to obtain a region score 144A for each candidate unigram 121A.
[0131] As detailed herein, the greedy decoder 120 may emit a metadata element 120A', such as confidence level metadata 120A', for each decoded unigram 120A. According to some embodiments, the unigram extraction module 140 may include a confidence score module 149 adapted to receive a confidence score 149A for each candidate unigram 121A of the greedy decoded corpus of unigrams 120A. The confidence score 149A may represent, for example, an average confidence of each unigram. This may be calculated, for example, by averaging all occurrences of the unigrams 120A in the greedy decoded output and calculating the average of their confidence level metadata 120A'.
[0132] As is known in the field of natural language processing, a word embedding vector may be a vector representation of the semantic meaning of a word, such that words that are closer in a vector space are expected to be similar in meaning. A word embedding vector may be obtained or generated by an embedding model (e.g., an ML-based model) based on a predefined corpus of words or unigrams.
[0133] According to some embodiments, the unigram extraction module 140 may generate multiple pair combinations 122 from the candidate unigrams 121A (or 121A'), where each pair 122 includes two candidate unigrams 121A and each pair is associated with one or more pair metric values 122A that define a relationship between the members of the candidate unigram pair 122.
[0134] As described in detail herein, the embedding model (e.g., word2vec model) 142 may generate a first word embedding vector 142A for at least one first candidate unigram 121A (or 121A') based on a corpus of candidate unigrams 121A (or 121A'), and may generate a second word embedding vector 142A for at least one second candidate unigram 121A (or 121A') based on the corpus of candidate unigrams 121A (or 121A').
[0135] According to some embodiments, the embedding model 142 may calculate, for one or more (e.g., each) pair 122 of the candidate unigram 121A (or 121A′), a pair metric value 122A that is a similarity score 142B based on the first unigram embedding vector 142A and the second unigram embedding vector 142A. For example, the similarity score 142B may be calculated as a cosine similarity between the two members of the pair of the candidate unigram 121A. In some embodiments, the embedding model 142 may generate only the embedding vector 142A and / or calculate the similarity score 142B based on the confidence score 149A (e.g., for the candidate unigram 121A having a confidence score 149A above a predefined threshold).
[0136] Additionally or alternatively, the unigram extraction module 140 may calculate a pair metric value 122A, which is a Levenshtein distance value, for one or more (e.g., each) pairs 122 of the candidate unigrams 121A (or 121A'), as described in detail herein (e.g., in connection with FIG. 5).
[0137] Additionally or alternatively, the unigram extraction module 140 may calculate, for one or more (e.g., each) pairs 122 of the candidate unigram 121A (or 121A'), a pair metric value 122A, which is the unigram frequency of occurrence of each member unigram 121A in the pair 122.
[0138] Additionally or alternatively, the unigram extraction module 140 may calculate, for one or more (e.g., each) pair 122 of a candidate unigram 121A, a pair metric value 122A that is an intra-domain score 144C for each unigram member 121A in the pair 122. As described in more detail herein, the unigram extraction module 140 may calculate an intra-domain score 144C based on a plurality of document unigrams 40A, such that the intra-domain score 144C may represent a likelihood that a candidate unigram is associated with at least one particular domain.
[0139] Additionally or alternatively, the unigram extraction module 140 may calculate, for one or more (e.g., each) pairs 122 of a candidate unigram 121A, a pair metric value 122A that is a confidence score (e.g., an average confidence level) for each unigram member 121A in the pair 122.
[0140] According to some embodiments, the unigram extraction module 140 may include a candidate list generator module 145 adapted to filter out the candidate unigrams 121A (or 121A') based on the pairwise metric values 122A and / or the respective unigram metadata 120A' to generate an initial list of candidate unigrams 145A. In other words, the initial list of candidate unigrams 145A may include the candidate unigrams 121A obtained by the greedy decoder 120, except for the filtered out candidate unigrams 121A, as detailed herein.
[0141] For example, the list generator module 145 may filter out unigram candidates 121A that have confidence scores 149A below a predefined confidence threshold. Additionally or alternatively, the list generator module 145 may filter out unigram candidates 121A that have occurrence counts or frequencies in the speech data elements 20 below a predefined threshold. Additionally or alternatively, the list generator module 145 may filter out unigram candidates 121A that do not appear in the corpus of document unigrams 40A. Additionally or alternatively, the list generator module 145 may filter out unigram candidates 121A that are in the vocabulary (e.g., included in the vocabulary of the language model 160A).
[0142] According to some embodiments, the unigram extraction module 140 may include one or more misspelling filters 146 configured to filter out unigram candidates 121A from the candidate unigram list 145A, as described in more detail herein.
[0143] The first misspelling filter 146 may be a common word misspelling filter 146 in the sense that it may be configured to filter out candidate unigrams 121A from the candidate unigram list 145A as likely misspellings of common words (e.g., non-domain words having an in-domain score 144C below a predefined threshold).
[0144] As detailed herein, common (e.g., non-domain specific) words such as "is", "me", "I", "have", "why" may have the same frequency in all call center domains (fashion, telecommunications, pharmaceuticals, etc.). In detailed embodiments, the intra-domain score 144C may be normalized by dividing the frequency of occurrence of related words in the intra-domain corpus by their frequency of occurrence in the background corpus. Thus, the intra-domain score 144C of common words may have a value close to 1.0.
[0145] According to some embodiments, the common word misspelling filter 146 may remove or filter out first candidate unigrams 121A that have at least one second paired unigram (e.g., pair 122), where the second candidate unigrams 121A (a) have an in-domain score 144C that is within a predefined range value (e.g., suspected to be a common word) or have a confidence score (e.g., average confidence level) above a predefined threshold, and (b) have a Levenshtein distance value below a predefined threshold to the first candidate unigram 121A.
[0146] In another example, the common word misspelling filter 146 may remove or filter out a first unigram candidate 121A that has at least one second paired unigram (e.g., pair 122), where the second candidate unigram 121A has a higher frequency of occurrence in the speech data element 20 than the first unigram candidate 121A.
[0147] Additionally, the common word misspelling filter 146 may store one or more unigram candidates 121A candidate unigram lists 145A based on language-specific rules. For example, the common word misspelling filter 146 may store unigram candidates 121A candidate unigrams that include multiple conjugations (e.g., concert vs. concerts).
[0148] The second misspelling filter 146 may be an in-domain word misspelling filter 146 in the sense that it may be configured to filter out candidate unigrams 121A from the candidate unigram list 145A as likely misspellings of in-domain words (e.g., unigrams having an in-domain score 144C above a predefined threshold).
[0149] For example, the within-domain word misspelling filter 146 may remove or filter out a first unigram candidate 121A having at least one second paired unigram (e.g., pair 122) where (a) a Levenshtein distance value between members of the pair of candidate unigrams 121A is below a predefined threshold, (b) the second candidate unigram 121A has a confidence score (e.g., average confidence level) above a predefined threshold, and (c) the second candidate unigram 121A has a higher frequency of occurrence (e.g., by a predefined factor) in the speech data elements 20 than the first unigram candidate 121A.
[0150] According to some embodiments, the spelling filter 146 may calculate a misspelling probability 146A of one or more candidate unigrams 121A based on one or more pairwise metric values 122A. For example, the spelling filter 146 may calculate the misspelling probability 146A as a function (e.g., a weighted sum) of the Levenshtein distance value 133A, the embedding similarity score 142B, the intra-region score 144C, and / or the frequency of occurrence 144B. The spelling filter 146 may then filter out candidate unigrams 121A whose misspelling probability 146A exceeds a predefined threshold.
[0151] According to some embodiments, the unigram extraction module 140 may include a context vector generator module 147 configured to generate a context vector 147A for each first candidate unigram 121A of the remaining candidate unigrams 121A (e.g., after filtering by the candidate list generator 145 and the spelling mistake filter 146).
[0152] For example, for each particular remaining candidate unigram 121A (e.g., remaining after the previous steps of the list generator 145 and the spelling error filter 146), the context vector generator module 147 may prepare a first intermediate list of unigrams taken from the union of the greedy decoder transcription unigrams 121A resulting from the speech data element 20 including the candidate unigram 121A and the in-vocabulary words (e.g., unigrams already present in the language model 160A). The context vector generator module 147 may then remove from the first intermediate list all unigrams having an in-domain score 144C below a predefined threshold to generate a second intermediate list. The context vector generator module 147 may prepare a third intermediate list of all unigrams that appear in the same speech data element 20 as the particular candidate unigram 121A. The context vector generator module 147 may then intersect the second intermediate list with the third intermediate list and drop duplicates. The result of this intersection may be referred to herein as a context vector 147A for a particular candidate unigram 121A.
[0153] It can be appreciated that the context vector 147A can be a list that includes multiple unigrams that are associated with the same context of a particular candidate unigram 121A.
[0154] For example, experimental application of system 100 has shown that candidate unigram 121A "rodeo" may have a context vector 147A that may include unigrams such as "chicago," "club," "concert," "cowboy," "game," "venue," "featuring," "garden," etc. Each unigram of context vector 147A has a domain score above the predefined threshold discussed above.
[0155] In another experimental example from the sports subject domain, candidate unigram 121A “dodgers” yielded context vector 147A that included “angels”, “baseball”, “boston”, “chicago”, “cubs”, “game”, “fame”, “reds”, “season”, “stadium”, “yankees”, etc.
[0156] In another experimental example from the pharmaceutical subject domain, candidate unigram 121A “injection” yielded context vector 147A including “absorbed”, “administer”, “blood”, “dose”, “drug”, “needle”, “puncture”, “medicine”, “substance”, “syringes”, “localized”, etc.
[0157] According to some embodiments, the unigram extraction module 140 may include a correctness score calculation module 148. The correctness score calculation module 148 may be adapted to calculate a correctness score 148A for each of the remaining candidate unigrams 121A (e.g., after filtering of the candidate list generator 145 and the spelling mistake filter 146) based on the candidate unigram's respective context vector 147A.
[0158] For example, for each particular candidate unigram 121A, the accuracy score computation module 148 may iterate over all documents (eg, Wikipedia summaries) in the document corpus 40 that contain the particular candidate unigram 121A.
[0159] In each iteration, the accuracy score calculation module 148 may intersect the set of unique unigrams 40A from the relevant documents 40 with the context vector 147A of the candidate unigram 121A. As described herein, the context vector or context list 147A may include a subset of document unigrams 40A that (a) have a domain score 144C that exceeds a predefined threshold and (b) appear in one or more speech data elements 20A (e.g., from which a particular candidate unigram 121A was obtained).
[0160] In other words, for each document 40, the accuracy score calculation module 148 may obtain an intersection group that may include unigrams that appear in the document and in the context list or vector 147A. The accuracy score calculation module 148 may then calculate, for each document, an accuracy score that represents the relevance of the candidate unigram to the document based on the intra-domain scores 144C of the document unigrams in the intersection group.
[0161] For example, the accuracy score calculation module 148 may calculate the accuracy score 148A by summing the intra-region scores 144C of the intersected unigrams. Additionally, the accuracy score calculation module 148 may normalize the calculated accuracy score 148A, for example, by multiplying the accuracy score 148A by the square root of the number of unigrams in the intersection (e.g., 40A). Throughout the iterations, the accuracy score calculation module 148 may retain the highest or maximum accuracy score 148A value (e.g., the highest among all documents 40) and the identity of the corresponding document 40.
[0162] According to some embodiments, the accuracy score calculation module 148 may proceed to find a maximum accuracy score 148A for one or more (e.g., each) of the remaining candidate unigrams 121A and corresponding documents 40. The accuracy score calculation module 148 may then filter out (e.g., exclude) the candidate unigrams 121A that correspond to a maximum accuracy score below a predefined threshold.
[0163] For example, the correctness score calculation module 148 may sort the candidate unigrams 121A in descending order based on their correctness scores 148A and remove tails below a certain threshold. It can be appreciated that this sorted list of candidate unigrams, each of which appears in the best context match documents 40, may provide an indication of the correctness of the word.
[0164] For example, experimental results of candidate unigram 121A evaluation in the biopharmaceutical subject area produced the following list of candidate unigrams 121A along with their corresponding accuracy scores 148A: "prescribed" - 19628, "ingredient" - 15540, "dose" - 15042, "prostate" - 10541, "syringe" - 8681, "antibiotics" - 8284, "anxiety" - 8477, "kidneys" - 8046, etc.
[0165] In another example, experimental results of candidate unigram 121A evaluation in the subject domain of reselling tickets (e.g., tickets to sporting events, theaters, operas, etc.) yielded the following list of candidate unigrams 121A with corresponding accuracy scores 148A: "performances" - 5439, "yankees" - 4471, "cubs" - 3803, "angels" - 3803, "stadium" - 3803, "reds" - 3481, "rodeo" - 3269, "dodgers" - 2574, "braves" - 2511, "concerts" - 2412, "playoff" - 2182, "nationals" - 1869, etc.
[0166] The unigram extraction module 140 may select or identify the top scoring candidate unigrams 121A (e.g., having the highest accuracy scores 148A). These unigrams are referred to herein as identified unigrams 140A. The identified unigrams 140A (similar to identified unigrams 130A) may then be further analyzed to improve the language model 160A, as described in more detail herein.
[0167] Reference is now made again to Figure 4. According to some embodiments, and as detailed herein, for example in connection with Figures 3 and / or 4, the system 100 may include an n-gram expansion module 150. The n-gram expansion module 150 may be configured to analyze the audio data element 20, taking into account the identified unigrams 130A and / or 140A, to extract at least one n-gram 150A from the audio data element 20. The at least one n-gram 150A may include one or more of the identified unigrams 130A / 140A.
[0168] As described in more detail herein, the greedy decoder 120 may compute a unigram metadata 120A' element that represents a confidence level or confidence score for one or more (eg, each) generated unigrams.
[0169] According to some embodiments, the n-gram expansion module 150 may analyze occurrences of the unigram 120A in the greedy decoder transcription. After identifying one or more candidate unigrams 130A / 140A, the n-gram expansion module 150 may locate, in the speech data element 20, n-grams of adjacent unigrams that include the candidate unigram 130A / 140A.
[0170] According to some embodiments, for one or more candidate unigrams 130A / 140A, the n-gram expansion module 150 may locate n-grams of adjacent unigrams that include the candidate unigram 130A / 140A in the audio data element 20. In other words, the n-gram expansion module 150 may search the greedy encoder 120 transcription of the audio data element 20 (which is the sequence of greedy decoded unigrams 120A) to find the identified unigrams 130A and / or 140A within the sequence of decoded unigrams 120A.
[0171] The N-gram expansion module 150 may use the identified unigrams 130A and / or 140A as "anchor" unigrams in an expanded search algorithm, as detailed herein. The term "anchor" may be used in this context to refer to an identified unigram (e.g., 130A and / or 140A) that may mark the start of an expanded search process.
[0172] The N-gram expansion module 150 may be configured to find sequences of unigrams 120A (e.g., words) that include the unigrams 130A and / or 140A identified in the speech data element 20. These sequences of unigrams 120A are referred to herein as extracted n-grams 150A. The N-gram expansion module 150 may then update the language model 160A to include at least one extracted n-gram 150A.
[0173] According to some embodiments, the N-gram expansion module 150, as indicated by its name, may perform a process of expansion search, starting from the anchor unigram 130 / 140, in the sequence of unigrams 120A obtained by the greedy encoder 120. The term "expansion" may be used herein to indicate that an n-gram may be constructed by expanding the sequence of unigrams 120A in at least one direction from the anchor unigram 130 / 140 until a stopping condition is met.
[0174] For example, starting from an anchor-identified unigram 130A / 140A in the sequence of unigrams 120A obtained from the greedy decoder 120, the expansion search can proceed to the next unigram (e.g., word) in both directions (e.g., to the previous unigram in the sequence of unigrams 120A and to the subsequent word in the sequence of unigrams 120A).
[0175] If (a) the next unigram 120A has a confidence value metadata 120A' above a predefined threshold, and (b) the next unigram 120A is also included in the group of identified unigrams 130A / 140A and / or included in the language model 160A, the next unigram 120A is added or merged into the expanded n-gram 150A. If the next unigram 120A does not satisfy the above conditions (a) and (b), the stop condition is met in the relevant direction and the expansion search process is stopped in that direction. If the stop condition is met in both directions, the expansion search process is terminated for the relevant anchor identified unigram 130A / 140A and the expansion of the n-gram 150A is stopped. The n-gram expansion module 150 may then proceed to the next anchor identified unigram 130A / 140A in the sequence of greedy decoded transcription unigrams 120A.
[0176] It can be seen that the expanded search process detailed above can ensure that all unigrams or words in n-gram 150A can have a confidence level 120A' greater than a predefined threshold and therefore are likely to be spelled correctly, which acts as an additional filter that increases the probability that n-gram 150A is correct.
[0177] Thus, according to some embodiments, if (a) a unigram 120A of expanded n-gram 150A corresponds to a confidence level 120A' that exceeds a predefined threshold and (b) expanded n-gram 150A includes at least a predefined threshold number of unigrams, language model builder 160 may update language model 160A to include at least one extracted n-gram 150A. Otherwise, candidate unigram 130A / 140A may be filtered from the corpus of candidate unigrams and may not be introduced into language model 160A.
[0178] According to some embodiments, N-gram expansion module 150 may limit the length of n-grams 150A to a maximum value or length in words. For example, N-gram expansion module 150 may limit the length of n-grams 150A to six words in each direction relative to the anchor unigram, resulting in a maximum n-gram length of 13 words (including the anchor unigram).
[0179] Additionally or alternatively, n-gram expansion module 150 may limit the length of n-gram 150A to a minimum value, such as three words in length. Such a restriction may ensure that n-gram 150A can maintain contextual relationships between member unigrams within the n-gram.
[0180] According to some embodiments, the N-gram expansion module 150 may limit the amount of verbatim identical n-grams it contributes to the language model, including extracted out-of-vocabulary unigrams.
[0181] According to some embodiments, in addition to extracting n-grams 150A "as is" from the sequence of greedy decoded unigrams 120A, the N-gram expansion module 150 may recover more n-grams by correcting unigram anchor occurrences 130A / 140A that are likely to be misspelled.
[0182] For example, the N-gram expansion module 150 may cooperate with the misspelling filter(s) 146 of FIG. 7 to prepare a list of unigrams 151A, which may include unigrams that are incorrectly misspelled, resulting in unigram anchors 130A / 140A.
[0183] The unigrams 151A may include unigrams that are similar to at least one anchor unigram 130A / 140A in the sense that, for example, their mutual embedding similarity scores 142B (e.g., word2vec similarity scores) are above a predefined threshold (e.g., 0.7) and their mutual Levenshtein distance values 133A are below a predefined threshold (e.g., 2). For one or more (e.g., each) anchor unigram occurrence 130A / 140A, the n-gram expansion module 150 may maintain a mapping 151B to the original correctly spelled unigram 151A. Once the n-grams 150A are extracted, the n-gram expansion module 150 may use the mapping 151B to replace the anchor unigram occurrence 130A / 140A with the correctly spelled unigram version 151A. If a potentially misspelled anchor unigram 130A / 140A corresponds to multiple correctly spelled unigram versions 151A, the n-gram expansion module 150 may select the one that corresponds to the highest mutual embedding similarity score 142B. For example, "ankees" is a likely misspelling of "yankees" because it has an embedding similarity score 142B (e.g., word2vec similarity score) of 0.853 and a Levenshtein distance value of 1.
[0184] Note that there may be cases of anchor unigram 130 / 140 splitting that, if not properly corrected, may result in erroneous n-gram 150A extractions. For example, a unigram such as "troubleshoot" may be split into "trouble" and "shoot." "Trouble" may function as a legitimate (e.g., correctly spelled) unigram anchor in itself and may be seen in other contexts (e.g., not necessarily followed by "shoot"), but it may still be an incorrect anchor.
[0185] According to some embodiments, to correct such cases of split unigrams, n-gram expansion module 150 may post-process extracted n-grams 151A. In this post-processing, n-gram expansion module 150 may fuse one or more (e.g., each) bigram 151C (e.g., consecutive unigrams) in n-gram 150A and analyze each fused bigram 151C separately. For example, a bigram may include consecutive unigrams "rec" and "tangle." The subsequent fused bigram 151C may be "rectangle." If the fused bigram 151C forms a legal (e.g., correctly spelled) unigram that is already included in an existing (e.g., language model 160A), n-gram 150A may be modified to include fused bigram 151C instead of the pair of pre-fused unigrams.
[0186] Note that there may be cases where the fusion of bigrams (e.g., two consecutive unigrams) may form a misspelled unigram anchor. In such conditions, the n-gram expansion module 150, in cooperation with the misspelling filter(s) 146 of FIG. 7, may add a step to the bigram fusion process to check whether the fused bigram 151C forms a misspelled anchor 130A / 140A. According to some embodiments, this is when adding collocation underlines, as detailed herein (e.g., in connection with the preprocessing module 141 of FIG. 7), may be utilized. The collocation underlines may be identified and inserted into a similarity table (e.g., Table 2). The misspelling filter(s) 146 may then remove the underline symbols and insert the result as an optional unigram for replacement 151A.
[0187] For example, the candidate unigrams 121A "reck" and "tangle" may be identified as collocations by the preprocessing module 141 and fused to obtain a consolidated candidate unigram "reck_tangle" 121A. The misspelling filter(s) 146 may insert the candidate unigram "reck_tangle" 121A into a similarity table (e.g., Table 2) with a similar unigram "rectangle" that may later be found as a unigram anchor 130A / 140A according to the following process. (a) spelling filter 146 may remove the underscore in “reck_tangle” to produce “recktangle”, (a) spelling filter(s) 146 may map 151B “recktangle” as associated with the correctly spelled unigram 151A “rectangle”, (c) n-gram expansion module 150 may post-process the n-gram containing “reck tangle” and fuse the two components to form the fused bigram 151C “recktangle”, (d) because “recktangle” is mapped 151B to the correctly spelled unigram form 150A “recktangle”, n-gram expansion module 150 may replace “recktangle” with “rectangle”, resulting in the correctly spelled n-gram 151A.
[0188] According to some embodiments, the n-gram expansion module 150 may add newly extracted n-grams 150A to an existing in-house text corpus of the acoustic transcriptions 20B. For example, the in-house acoustic transcriptions 20B may include call center transcriptions dedicated to discussions on a particular subject area (e.g., banking, health, sports, etc.), where they may be updated with newly extracted n-grams 150A including newly identified correctly spelled words.
[0189] Additionally, the n-gram extension module 150 may add the newly extracted n-grams 150A to the audio transcription 20B along with some weighted value 20B', which may represent a default priority. For example, the weight 20B' may refer to a date priority (e.g., the most recent discussion is weighted as more relevant than an earlier discussion). In another example, the weight 20B' may indicate a subject priority (e.g., emergency issues are weighted as more relevant than non-emergency issues). Other weighted values 20B' may also be implemented.
[0190] According to some embodiments, n-gram extension module 150 may cooperate with n-gram language model builder model 160 (or "builder 160" for short). Builder 160 may receive the newly extracted n-grams 150A from n-gram extension module 150 and may rebuild or recompile language model 160A to include the added newly extracted n-grams 150A. System 100 may then proceed to use beam decoder 170 with new language model 160A as detailed herein (e.g., in connection with FIG. 2) to generate transcription 100A of input speech data element 20A.
[0191] According to some embodiments, system 100 may utilize extracted n-grams 150A as training data to automatically retrain or fine-tune the training of acoustic model 110 based on speech data elements 20. In other words, system 100 may automatically fine-tune neural network 110 using a training data set that includes utterances of newly discovered words to facilitate future recognition of these words in speech data elements 20.
[0192] According to some embodiments, the greedy decoder 120 may generate, for each decoded unigram 120A, a metadata element that is a time stamp of the utterance of the unigram 120A in the audio data element 20. The N-gram expansion module 150 may generate at least one corresponding snippet time stamp 150B for one or more extracted n-grams 150A based on the time stamp metadata 120A'. For example, the snippet time stamp 150B may include a time stamp of the beginning of the n-gram 150A in the audio data element 20 (e.g., a time stamp 120A' of a first unigram 120A in the n-gram 150A). In another example, the snippet time stamp 150B may include a time stamp of the end of the n-gram 150A in the audio data element 20 (e.g., a time stamp 120A' of a last unigram 120A in the n-gram 150A).
[0193] The N-gram expansion module 150 may cooperate with an artificial neural network (ANN) refinement module 180. The ANN refinement module 180 may receive one or more extracted n-grams 150A and corresponding snippet time stamps 150B (e.g., start and end times) and may filter the extracted n-grams 150A to generate groups of n-grams 150A that are longer than a predefined period (e.g., 2.5 seconds in length).
[0194] The ANN refinement module 180 may include the extracted n-grams 150A (here, transcriptions of snippets of acoustic speech data elements 20) in an in-house corpus of acoustic transcriptions 20B. The ANN refinement module 180 may then retrain the acoustic model 110 based on the snippets of acoustic speech data elements 20 (defined by time stamps 150B) using the added extracted n-grams 150A as training data.
[0195] Additionally, the ANN refinement module 180 may attribute weights to the added extracted n-grams 150A to enhance retraining of the acoustic model 110 based on this new training data. The ANN refinement module 180 may fine-tune the training of the acoustic model 110 with this mixed data set for several epochs (e.g., 10). Experiments have shown that such training can significantly increase recall of words in the extracted n-grams 150A.
[0196] In addition, it has been experimentally shown that the overall word error rate gain is substantial even when stopping at the best improvement for the target group set. It remains significant (but less so) when taking the last model. During testing, the word error rate of a subset of utterances (e.g., utterances containing at least one of the unigram anchors) is lowered to nearly the same value as the overall word error rate, which was previously quite high. This suggests that out-of-vocabulary words are initially "harder" for the neural network to recognize correctly, but once introduced to the neural network, they become as "familiar" as the rest of the known vocabulary.
[0197] Reference is now made to FIG. 8, which is a flow diagram illustrating a method for automatically discovering unigrams within speech data elements by at least one processor of a system for speech recognition according to some embodiments of the present invention.
[0198] As shown in step S1005, at least one processor (e.g., element 2 in FIG. 1) may receive a language model 160A, which may include multiple n-grams. Each such n-gram may include one or more unigrams.
[0199] As shown in step S1010, at least one processor 2 may apply an acoustic machine learning model (e.g., acoustic model 110 of FIG. 2) to one or more first audio data elements (e.g., audio element 20 of FIG. 4) to obtain a character distribution function (e.g., feature distribution element 110A of FIG. 2).
[0200] As shown in steps S1015 and S1020, the at least one processor 2 may apply a greedy decoder (e.g., greedy decoder 120 of FIG. 4) to the character distribution function 110A to predict an initial corpus of unigrams (e.g., element 120A of FIG. 3). The at least one processor 2 may filter out one or more unigrams of the initial corpus 120A to obtain a corpus of candidate unigrams (e.g., elements 121A, 130A, 140A of FIG. 3) that are not included in the language model 160A.
[0201] As shown in step S1025, and as described in more detail herein (e.g., in connection with Figures 3-7), at least one processor 2 may analyze one or more first audio data elements 20 to extract at least one n-gram (e.g., element 150A in Figure 4) that includes a candidate unigram (e.g., 121A, 130A, 140A).
[0202] As shown in steps S1030 and 1035, the at least one processor 2 may update the language model 160A to include the extracted at least one n-gram 150A. The at least one processor 2 may then (e.g., in an inference stage) apply a beam decoder (e.g., beam decoder 170 of FIG. 2) to the at least one second audio data element 20 to generate at least one corresponding transcription of the second audio data element 20 based on the updated language model 160A.
[0203] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Furthermore, all formulas described herein are intended as examples only, and other or different formulas may be used. Additionally, some of the method embodiments described or elements thereof may occur or be performed at the same time.
[0204] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents may occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the invention.
[0205] Various embodiments have been presented, each of which may, of course, include features from the other embodiments presented, and embodiments not specifically described may include various features described herein.
Claims
1. 1. A method for automatically discovering unigrams in audio data elements by at least one processor, comprising: receiving a language model including a plurality of n-grams, each n-gram including one or more unigrams; applying an acoustic machine learning (ML) model to the one or more first speech data elements to obtain a character distribution function; applying a greedy decoder to the character distribution function to predict an initial corpus of unigrams; filtering out one or more unigrams of the initial corpus to obtain a corpus of candidate unigrams not included in the language model; analyzing the one or more first speech data elements to extract at least one n-gram including a candidate unigram; and updating the language model to include the extracted at least one n-gram.
2. 2. The method of claim 1, further comprising applying a beam decoder to the second speech data element to generate at least one transcription of the second speech data element based on the updated language model.
3. The method of claim 1 , wherein the character distribution function represents a likelihood of occurrence of one or more linguistic characters in the one or more first audio data elements.
4. 2. The method of claim 1, further comprising retraining the acoustic ML model based on at least one of the second speech data elements using the at least one extracted n-gram as training data.
5. Filtering out one or more unigrams calculating, for one or more candidate unigrams, a misspelling probability that represents the likelihood that an associated unigram is a misspelled version of a unigram already included in the language model; The method of claim 1 , further comprising filtering out candidate unigrams that correspond to a misspelling probability that exceeds a predefined threshold.
6. Calculating the probability of a spelling error calculating a Levenshtein distance value between the candidate unigram and at least one second unigram already included in the language model; calculating a frequency score representative of a rate of occurrence between the candidate unigram and the at least one second unigram in the one or more first speech data elements; and calculating the misspelling probability based on the Levenshtein distance value and the frequency score.
7. The greedy decoder is adapted to emit a respective confidence level for each unigram of the initial corpus, and calculating a misspelling probability comprises: calculating a confidence score representing an average of the confidence levels for one or more occurrences of the candidate unigram in the one or more first speech data elements; The method of claim 6 , further comprising: calculating the misspelling probability further based on the confidence score.
8. The language model further comprises a definition of one or more language syntax rules, and calculating the Levenshtein distance value comprises: calculating a number of single character edits between the candidate unigram and the second unigram; and calculating the Levenshtein distance value based on the one or more language syntax rules and the number of single-character edits.
9. calculating, for one or more candidate unigrams, a missing space probability that represents the likelihood that the candidate unigram is a concatenation of two unigrams already included in the language model; The method of claim 5 , further comprising filtering out candidate unigrams that correspond to missing space probabilities that exceed a predefined threshold.
10. computing a first unigram embedding vector for at least one first candidate unigram based on the corpus of candidate unigrams; calculating, for at least one second candidate unigram, a second unigram embedding vector based on the corpus of candidate unigrams; calculating a similarity score based on the first unigram embedding vector and the second unigram embedding vector; The method of claim 7 , further comprising: calculating the misspelling probability further based on the similarity score.
11. 11. The method of claim 10, further comprising receiving a document corpus comprising a plurality of documents, each document associated with a particular subject area, each document comprising a plurality of document unigrams.
12. 12. The method of claim 11, further comprising: calculating, for one or more candidate unigrams, an intra-region score based on the plurality of document unigrams, the intra-region score representing a likelihood that the candidate unigram is associated with at least one particular region.
13. For one or more candidate unigrams, compiling a context list comprising a subset of document unigrams that (a) have intra-domain scores above a predefined threshold and (b) appear in the one or more first speech data elements; for each document, obtaining an intersection group that includes unigrams that appear in the document and in the context list; calculating, for each document, a precision score representing the relevance of the candidate unigram to the document based on the intra-domain scores of document unigrams in the intersection group; 13. The method of claim 12, further comprising filtering out candidate unigrams corresponding to a maximum accuracy score below a predefined threshold.
14. The greedy decoder is adapted to issue a respective confidence level for each unigram of the initial corpus, and analyzing the speech data elements of the one or more first speech data elements comprises: for one or more candidate unigrams, locating within said speech data element an n-gram of adjacent unigrams that includes said candidate unigram; (a) the unigram of the n-gram corresponds to a confidence level exceeding a predetermined value; and (b) if the n-gram contains more than a predetermined threshold number of unigrams, the language model is updated to include the extracted at least one n-gram; If not, the candidate unigram is filtered out from the corpus of candidate unigrams.
15. 1. A system for automatically discovering unigrams in audio data elements, comprising: a non-transitory memory device having a module of instruction code stored therein; and at least one processor associated with said memory device and configured to execute said module of instruction code, wherein upon execution of said module of instruction code, said at least one processor: receiving a language model including a plurality of n-grams, each n-gram including one or more unigrams; applying an acoustic ML model to the one or more first speech data elements to obtain a character distribution function; applying a greedy decoder to the character distribution function to predict an initial corpus of unigrams; filtering out one or more unigrams of the initial corpus to obtain a corpus of candidate unigrams not included in the language model; analyzing the one or more first speech data elements to extract at least one n-gram including a candidate unigram; and updating the language model to include the extracted at least one n-gram.
16. 16. The system of claim 15, further comprising a beam decoder configured to receive at least one second audio data element and generate a transcription of the at least one second audio data element based on the updated language model.
17. The at least one processor: calculating, for one or more candidate unigrams, a misspelling probability that represents the likelihood that an associated unigram is a misspelled version of a unigram already included in the language model; and 16. The system of claim 15, configured to filter out one or more unigrams by filtering out candidate unigrams that correspond to a misspelling probability that exceeds a predefined threshold.
Citation Information
Patent Citations
Sentence input device by means of speech recognition
JP2002229585A
Voice recognition device and method, recording medium and program
JP2003186494A
Voice processor, voice processing method and program
JP2019101285A
