Improved speech recognition transcription

Abstract

Techniques can be provided for correcting transcriptions of speech recognition models. For a particular node associated with a virtual assistant, a list of similar-sounding phonemes associated with phonemes of high-frequency terms can be generated. The utterance can be transcribed and receive a confidence score for the accuracy of the transcription based on speech metrics and other factors. The phonemes of the utterance can be compared to the phonemes of the high-frequency terms in the list, and scores for matching and similar-sounding phonemes can be determined. If the similar-sounding score of a term in the high-frequency term list is determined to be above a threshold, the transcription can be replaced with that term, thereby providing a corrected transcription.

Description

[Technical Field]

[0001] The present invention relates generally to the field of speech recognition, and more particularly to improving transcription of speech utterances in speech recognition. [Background technology]

[0002] Speech recognition models have long attempted to enable users to interact with computing devices through speech or voice commands. The ability of voice assistants to process voice commands and utterances has evolved significantly over the past decade due to improvements in processing power and memory capacity. These improvements have enabled the development of new user interfaces that allow users to command computing devices through voice commands and utterances. In some models, speech recognition involves receiving sound waves, identifying phonemes from the sound waves, and assigning computer-understandable meaning to the phonemes. Summary of the Invention

[0003] Embodiments of the present disclosure include computer-implemented methods, computer program products, and systems for training a model to improve speech recognition for a voice assistant. Training a model to improve speech recognition may include receiving a history of utterances and corresponding audio metrics for the utterances, and generating a similar sound list for at least one utterance based on the history of the utterances and the audio metrics for the utterances.

[0004] The above summary is not intended to describe every example embodiment of every implementation of the present disclosure. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a functional block diagram that schematically illustrates a speech recognition transcription correction environment, according to one embodiment of the present invention. [Figure 2]FIG. 2 is a functional block diagram illustrating a transcription correction engine according to one embodiment of the present invention. [Figure 3] FIG. 2 is a functional block diagram illustrating an automatic speech recognition module according to one embodiment of the present invention. [Figure 4] 1 is a flowchart illustrating a method for generating a similar sound list according to one embodiment of the present invention. [Figure 5] 1 is a flowchart illustrating a method for correcting a speech recognition transcription according to one embodiment of the present invention. [Figure 6] FIG. 1 is a functional block diagram illustrating an exemplary computing system in a speech recognition transcription correction environment, according to one embodiment of the present invention. [Figure 7] FIG. 1 illustrates a cloud computing environment according to one embodiment of the present invention. [Figure 8] FIG. 2 is a functional block diagram illustrating abstraction model layers according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0006] While the embodiments described herein are susceptible to various modifications and alternative forms, specific details of the embodiments have been shown by way of example in the drawings and will be described in detail. It is to be understood, however, that the particular embodiments described are not to be construed in a limiting sense. On the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure.

[0007] The illustrated embodiment enables a technique for correcting speech recognition transcriptions, and more specifically, a technique for comparing phonemes in low confidence transcriptions with phonemes in expected high frequency terms and correcting the low confidence transcriptions based on the comparison.

[0008] In one embodiment of the present invention, a log of recorded user utterances and speech metrics are received at a particular node of a virtual assistant (VA). The VA can be a question / answer program (e.g., International Business Machines Corporation's Watson®, Apple Inc.'s Siri®, Amazon LLC's Alexa®, etc.), or the VA can be a computer program associated with a user being given prompts and responding to the prompts with utterances or commands, such as experienced when calling a customer service number. Additionally, a node can be a single prompt or question given by the VA, with the VA expecting a specific response. Recordings are identified based on terms spoken by the user as determined by the VA. The most frequent utterances are also identified. The most frequent utterances are extracted as their respective phonemes. The extracted phonemes can be separated to generate a list of phonemes that are phonetically similar to the phonemes. A similarity confidence score can be generated for the phonemes in the list as to how phonemes are phonetically similar to the extracted phoneme. This similar sound list can be stored in a data repository corresponding to each VA node.

[0009] In another embodiment of the present invention, a speech recording can be received at a node based on a prompt from a VA. The speech can be transcribed by an Automatic Speech Recognition (ASR) module, and a confidence score for the transcription can be assigned based on the expected response to the VA node's prompt, audio metrics of the recording, or both. If the confidence score is below a threshold, the transcription is considered a "miss." Good candidate transcriptions from the node's "sounds like" list of high-frequency term responses can be retrieved from a data repository. The miss transcription can be matched to a good transcription based on the expected high-frequency terms, and the phonemes of the "miss" can be compared to the phonemes of the candidate good transcription. The matched transcription is scored based on properly aligned phonemes. If the matched score exceeds a threshold, the "miss" transcription can be replaced with the highest-scoring candidate good transcription.

[0010] 1 is a functional block diagram that schematically illustrates a speech recognition transcription correction environment 100. The speech recognition transcription correction environment 100 includes an automatic speech recognition (ASR) module 104 and a transcription correction engine 106 running on a server 102, a data repository 108 stored on the server 102, a client computer 112, and a network 110 that supports communication between the server 102 and the client computer 112. While only the server 102 is shown for simplicity, it should be noted that the environment may include multiple (i.e., 1, 2, n...n+1) servers and other computing devices accessible via the network 110.

[0011] The server 102 and the client computers 112 can be standalone computing devices, management servers, web servers, mobile computing devices, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In other embodiments, the server 102 and the client computers 112 can correspond to a server computing system that uses multiple computers as a server system. In another embodiment, the server 102 and the client computers 112 can be laptop computers, tablet computers, netbook computers, personal computers, desktop computers, or any programmable electronic device that can communicate with other computing devices (not shown) in the speech recognition transcription correction environment 100 via the network 110.

[0012] In another embodiment, the server 102 and the client computers 112 represent a computing system that uses clustered computers and components (e.g., database server computers, application server computers, etc.) that can act as a single pool of seamless resources when accessed within the speech recognition transcription correction environment 100. The server 102 and the client computers 112 can include internal and external hardware components such as those illustrated and further detailed in connection with FIG.

[0013] The automatic speech recognition (ASR) module 104 may be a computer module capable of receiving speech or commands and converting them into a computer-readable format (described in more detail below). While one ASR module 104 is shown in Figure 1 running on the server 102, it should be noted that the ASR module 104 may run on any computing device that communicates with the transcription correction engine 106 over the network 110, or on a local computing device that has the transcription correction engine 106.

[0014] The transcription correction engine 106 may be a module that receives historical data logs. The historical data logs may include recordings of user utterances. For example, the recordings may be recordings of user utterances associated with one or more nodes in the virtual assistant system. The historical data logs may also include audio metrics corresponding to the recorded user utterances. The audio metrics may include information about the quality of the recording, including signal-to-noise ratio, background noise, speech ratio, high-frequency loss, DC offset, clipping ratio, speech level, and non-speech level. The audio metrics may be provided by software that extracts audio metric features, including, but not limited to, International Business Machines Corporation's Watson® Speech-to-Text service. Furthermore, the transcription correction engine 106 may be able to identify the most frequent terms from the historical data logs within a certain time frame (e.g., one month, two weeks, or a user-defined time frame). The transcription correction engine 106 may also separate the user utterances of the most frequent terms into the phonemes of the terms. A list of phonemes that sound similar to the isolated phoneme can be generated by the transcription correction engine 106 (discussed in more detail below).

[0015] Additionally, for a given VA node, the transcription correction engine 106 can receive a user utterance and audio metrics for the utterance, either recorded or in real time, and convert the utterance into a transcription. A transcription confidence score can be generated for the transcription based on the expected response to the utterance and the audio metrics. The transcription correction engine 106 can also correct the transcription based on a similar sound list (discussed in more detail below). Note that while FIG. 1 illustrates the transcription correction engine 106 operating on only one computing device, in some embodiments, the transcription correction engine 106 can operate on one or more computing devices or within a cloud computing system. The transcription correction engine 106 can perform some of the actions described above on the same computing device or on different computing devices.

[0016] Data repository 108 may be a database that can store data for a given VA node, including, but not limited to, generated "similar sound lists," phoneme confidence scores, transcription confidence scores, utterances, and corresponding phonetic metrics. Note that while Figure 1 shows data repository 108 located on server 102, in some embodiments, data repository 108 may be located on one or more computing devices or within a cloud computing system.

[0017] The network 110 may be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and may include wired, wireless, or fiber optic connections. In general, the network 110 may be any combination of connections and protocols that support communication between the server 102 and the client computers 112.

[0018] 2 is a functional block diagram 200 illustrating the transcription correction engine 106. A term identification module 202 and a phoneme comparison module 204 are shown operating within the transcription correction engine 106.

[0019] The term identification module 202 is a computer module that can receive or retrieve phoneme-decomposed utterances and speech metrics from the data repository 108. Additionally, the term identification module 202 can receive real-time phoneme-decomposed user utterances and speech metrics from the ASR module 104. In some embodiments, the term identification module 202 can identify high-frequency utterances and corresponding terms from a speech history log for a particular node of the VA. The term identification module 202 can also determine the rate at which a term is used at a given node over a period of time. The period of time can be static or dynamic based on configuration. For example, if, over a given month, a customer service VA at a financial institution provides a user with four prompts to choose from: 1) checking, 2) savings, 3) pension, or 4) loan, the term identification module 202 can determine which historically recorded utterances correspond to each term and generate the percentage of each term selected by the user (e.g., 55% checking, 25% savings, 10% pension, and 10% loan). Additionally, the term identification module 202 can identify utterances that do not exactly match due to poor speech metrics, accent, or incorrect pronunciation (e.g., a user saying "refirernent" instead of "retirement" or "savings" instead of "savings"). The term identification module 202 can identify high-frequency terms and their corresponding utterances and send the terms to the phoneme comparison module 204.

[0020] The phoneme comparison module 204 may be a computer module capable of analyzing phonemes extracted from speech history and real-time user utterances. In one embodiment, the phoneme comparison module may receive or retrieve high-frequency terms of a VA node and phonemes of the utterance from the term identification module 202. The phoneme comparison module 204 may also generate a similar-sounding list of phonemes of the utterance based on the terms of the VA node. For example, English has 44 phonemes. The phoneme comparison module 204 may determine the phonemes of a term and find phonemes similar to the term to create a "similar-sounding" list of phonemes. In some embodiments, each phoneme in the list may be assigned a similarity score. The similarity score may be based on a human-annotated phoneme list or an analysis of the similarity of speech spectrograms between two phonemes. In some embodiments, the similarity score may include regional dialect features based on an identification of the accent of the user speaking the utterance.

[0021] In some embodiments, the phoneme comparison module 204 may receive from the ASR module 104 a real-time transcription of a user utterance and extracted phonemes from that utterance that have a transcription confidence score below a threshold. The phoneme comparison module 204 may analyze phonemes from the utterance to generate candidate “good” transcriptions of the real-time user utterance. In some embodiments, the number of phonemes may be matched against the expected number of phonemes of the expected terms of the VA node. Terms with more or fewer phonemes may be rejected. For example, if a user attempts to say “au-thor-i-za-tion” and the ASR module 104 transcribes it as “of-her-i-say-shun,” the phoneme comparison module 204 will recognize that there are five phonemes in the utterance and will eliminate from consideration high-frequency terms in the list that have four or fewer phonemes and six or more phonemes. The remaining terms may also be analyzed to determine which phonemes match. Unmatched phonemes are compared to phonemes in the expected term's "similar sounds" list. For example, "i-say-shun" above is a match in the similar sounds list. The remaining two phonemes, "of" and "her," are compared to similar-sounding phonemes in the authorization's similar sounds list. In this example, "of" is in the similar sounds list for that phoneme and "au." However, "her" is not in the similar sounds list for the phoneme corresponding to "thor." If a match is found, the term receives a phoneme alignment score; if the phoneme alignment score exceeds a threshold (either predetermined or dynamic based on phonetic metrics), the newly aligned phonemes can replace the original transcription. The phoneme alignment score can be calculated by determining the percentage of correctly aligned phonemes in the original transcription and factoring that percentage by the similar sounds score for the misaligned phonemes. The new transcription can then be used to update the ASR module 104.

[0022] 3 is a functional block diagram 300 of the ASR module 104 according to one embodiment of the present invention. The ASR module 104 may comprise a speech translator 302, a speech transcriber 304, a tokenizer 306, a part-of-speech (POS) tagger 308, a semantic relation identifier 310, and a syntactic relation identifier 312. In some embodiments, the ASR module 104 may be a neural network or a hidden Markov model, or a combined neural network / hidden Markov model, capable of receiving an utterance, extracting phonemes from the utterance, and transcribing text from the extracted phonemes.

[0023] The speech translator 302 can receive or extract speech from a user. The speech can be recorded or received in real time as a sound wave model. The speech translator 302 can convert the sound wave model into a speech spectrogram for further analysis. The speech spectrogram can provide a visual representation of the duration, amplitude, and frequency of the speech in a two-dimensional representation. The speech translator 302 can determine phonemes based on an analysis of the speech spectrogram. Additionally, the spectrogram can be decomposed into smaller time frames (e.g., 10 milliseconds) to improve the determination of phonemes in the speech.

[0024] The speech transcriber 304 is a computer module capable of generating text based on analysis of phonemes received or retrieved from the speech translator 302. Some embodiments of the present invention may be capable of determining words based on a predictive model in which one or more previous phonemes are considered in the prediction. The speech transcriber may also accept input from the tokenizer 306, POS tagger 308, semantic relation identifier 310, and syntactic relation identifier 312 in creating the transcribed text. The speech transcriber may also provide functionality for assigning a transcription confidence score to the transcription based on speech metrics corresponding to the speech. In some embodiments, the transcription confidence score may be an assessment of signal-to-noise ratio, background noise, speech ratio, high frequency loss, DC offset, clipping ratio, speech level, and non-speech level. In other embodiments, the confidence score may be context-driven, where the score is based on an expected response for a particular VA node. The expected response may be the degree to which the transcription matches the expected response.

[0025] In some embodiments, the tokenizer 306 may be a computer module that performs lexical analysis. The tokenizer 306 may convert a string of characters into a series of tokens. A token may be a string of characters contained within a recording that can be classified as a meaningful symbol. In some embodiments, the tokenizer 306 may also identify word boundaries within a recording and break down any text in a corpus into constituent text elements, such as words, multi-word tokens, numbers, and punctuation. In some embodiments, the tokenizer 306 may receive a string of characters, identify lexemes within the string, and classify the lexemes into tokens.

[0026] Consistent with various embodiments, the POS tagger 308 may be a computer module that assigns words in a transcription to correspond to particular parts of speech. The POS tagger 308 may analyze a transcription of an utterance and assign a part of speech to each word or other token. The POS tagger 308 may determine the part of speech corresponding to a word based on the word's definition and the word's context. The word's context may be based on the word's relationship to adjacent and related words within a phrase, sentence, or paragraph. In some embodiments, the word's context may depend on one or more previously analyzed words in a corpus. Examples of parts of speech that can be assigned to a word include, but are not limited to, nouns, verbs, adjectives, adverbs, etc. Examples of other part-of-speech categories that the POS tagger 308 can assign include, but are not limited to, comparative or superlative adverbs, relative adverbs, conjunctions, determiners, negative particles, possessive markers, prepositions, relative pronouns, etc. In some embodiments, the POS tagger 308 may tag tokens of "an" words in the corpus with part-of-speech categories or other annotations. In some embodiments, the POS tagger 308 may tag tokens or words in the corpus as being parsed by the speech transcriber 304.

[0027] In some embodiments, the semantic relation identifier 310 may be a computer module that may be configured to identify semantic relations between recognized entities (e.g., words, phrases, images, etc.) in the corpus. In some embodiments, the semantic relation identifier 310 may determine functional dependencies between entities and other semantic relations in the transcription.

[0028] Consistent with various embodiments, the syntactic relation identifier 312 may be a computer module configurable to identify syntactic relations within a corpus of tokens. The syntactic relation identifier 312 may determine the grammatical structure of a sentence, such as which word groups are related as phrases and which words are subjects or objects of verbs. The syntactic relation identifier 312 may follow a formal grammar.

[0029] 4 is a flow chart illustrating a method 400 for generating a "similar sounds" list. At 402, a speech history and corresponding phonetic metrics are received at the transcription correction engine 106.

[0030] At 404, the transcription correction engine 106 identifies high-frequency terms from the speech history and speech metrics. High-frequency terms can be identified by the term identification module 202 by analyzing the number of times a term is selected in a VA node within a certain time frame. In some embodiments, the speech metrics can also be term identification information evaluated to determine whether a term is selected, and if an utterance has poor speech metrics, the utterance can be excluded from the final analysis.

[0031] At 406, the ASR module 104 can decompose the identified high-frequency terms into corresponding phonemes. In some embodiments, the ASR module 104 evaluates a sound wave model of the pre-recorded speech, and the speech translator 302 converts the sound wave model into a speech spectrogram and separates the phonemes from the speech spectrogram. The speech transcriber 304 can convert the separated phonemes into text using inputs from the tokenizer 306, the POS tagger 308, the semantic relation identifier 310, and the syntactic relation identifier 312, thereby ensuring that the transcribed text is semantically and syntactically correct.

[0032] At 408, the phoneme comparison module 204 generates a "similar sound" list of phonemes of the high-frequency term transcribed by the ASR module 104. The phoneme comparison module 204 can also assign a confidence score to the list of similar-sounding phonemes for each phoneme in the high-frequency term.

[0033] 5 is a flow chart illustrating a method 500 for correcting a transcription using a similar sound list. At 502, utterances and speech metrics can be received by the ASR module 104. The utterances can be from a particular VA node or within the VA's open dialogue framework. In some embodiments, the utterances can be from an automated customer phone support line within a particular context. In yet other embodiments, the utterances can be from an open dialogue framework where the VA is activated by a specific utterance, giving a pre-existing command or asking a question within a particular domain.

[0034] At 504, the ASR module 104 may transcribe the received utterance. In some embodiments, the ASR module 104 may break down the utterance into phonemes and generate text based on the phonemes. In some embodiments, the text may be generated using a predictive model, where the predictive model is a deep neural network.

[0035] At 506, the ASR module 104 can assign a transcription confidence score to the transcription of the utterance. In some embodiments, the transcription confidence score can be based on an evaluation of speech metrics, or context based on the expected response, or a combination of the two.

[0036] At 508, the transcription correction engine 106 determines whether the transcription confidence score exceeds a threshold. The threshold can be static or dynamic. If the threshold is static, it can be set by a user based on the user's judgment or current need. The threshold can be dynamically set based on many factors, including the currently available computing resources, the length of the utterance, the utilization of the VA node at the time the utterance is received, etc. If the transcription confidence score is below the threshold, the transcription is considered a "miss" and is passed on for further processing. If the confidence score is above the threshold, the transcription method ends.

[0037] At 510, the phoneme comparison module 204 compares phonemes in the "miss" transcription with phonemes in the high-frequency term for the given node. In some embodiments, the phoneme comparison module can be configured to analyze the number of phonemes in the transcription to determine which high-frequency terms contain the same number of phonemes. The phoneme comparison module 204 can also determine whether any phonemes in the transcription match with phonemes in the high-frequency term. Any phonemes in the high-frequency term that do not match with phonemes in the transcription can be compared with phonemes in the "similar sounds" list to determine whether the phonemes can be matched. The phoneme comparison module 204 can also assign a score to the matched phonemes and the number of matched "similar sounding" phonemes. The score can be a function of the percentage of matched phonemes and the number of matched "similar sounding" phonemes.

[0038] At 512, if the score of the matched phoneme is above a threshold (statically or dynamically specified), the "miss" transcription is replaced with a transcription corresponding to the matched phoneme. The newly created transcription can be used to update the ASR module 104.

[0039] At 514, the method ends.

[0040] 6 illustrates computer system 600, an exemplary computer system representative of server 102 and data repository 108 or any other computing device in one embodiment of the present invention. Computer system 600 includes a communications fabric 12 that provides communication between computer processors 14, memory 16, persistent storage 18, network adapters 28, and input / output (I / O) interfaces 26. Communications fabric 12 can be implemented by any architecture designed for passing data and / or control information between processors (e.g., microprocessors, communications and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 12 can be implemented by one or more buses.

[0041] Computer system 600 includes processor 14, cache 22, memory 16, network adapter 28, input / output (I / O) interface 26, and communications fabric 12. Communications fabric 12 provides communication between cache 22, memory 16, persistent storage 18, network adapter 28, and input / output (I / O) interface 26. Communications fabric 12 can be implemented by any architecture designed to pass data and / or control information between processors (e.g., microprocessors, communications and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 12 can be implemented by one or more buses or crossbar switches.

[0042] Memory 16 and persistent storage 18 are computer-readable storage media. In this embodiment, memory 16 includes persistent storage 18, random access memory (RAM) 20, cache 22, and program modules 24. Generally, memory 16 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 22 is a high-speed memory that improves performance of processor 14 by retaining recently accessed data and data near the recently accessed data from memory 16. As further illustrated and described below, memory 16 may include at least one of program modules 24 configured to implement the functionality of embodiments of the present invention.

[0043] Programs / utilities having at least one program module 24, including, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data, may be stored in memory 16. Each of the operating system, one or more application programs, other program modules, program data, or any combination thereof may comprise an implementation of a networking environment. The program module 24 generally implements the functions and / or methods of embodiments of the present invention described herein.

[0044] Program instructions and data used to implement embodiments of the present invention may be stored in persistent storage 18 and memory 16 for execution by one or more of the respective processors 14 via cache 22. In one embodiment, persistent storage 18 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 18 may include a solid-state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0045] The media used by persistent storage 18 may be removable. For example, a removable hard drive may be used as persistent storage 18. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 18.

[0046] Network adapter 28, in these examples, provides for communication with other data processing systems or devices. In these examples, network adapter 28 includes one or more network interface cards. Network adapter 28 may provide for communication through the use of physical or wireless communication links, or both. Program instructions and data used to implement embodiments of the present invention may be downloaded to persistent storage 18 via network adapter 28.

[0047] The I / O interface 26 allows for the input and output of data to and from other devices connectable to each computer system. For example, the I / O interface 26 may provide a connection to an external device 30, such as a keyboard, keypad, touch screen, or any other suitable input device or combination thereof. The external device 30 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to implement embodiments of the present invention may be stored on such portable computer-readable storage media and loaded into persistent storage 18 via the I / O interface 26. The I / O interface 26 also connects to a display 32.

[0048] Display 32 provides a mechanism for displaying data to a user and may be, for example, a computer monitor or a virtual graphical user interface.

[0049] Components described herein are identified based on the applications in which they are implemented in particular embodiments of the invention, but it should be understood that the names of any particular components herein are used merely for convenience, and therefore the invention should not be limited to use in any particular application identified and / or implied by such names.

[0050] The present invention may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions stored thereon for causing a processor to implement aspects of the present invention.

[0051] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with instructions recorded on them, and any suitable combination thereof. As used herein, computer-readable storage media should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through fiber optic cable), or electrical signals transmitted over wires.

[0052] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0053] Computer-readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or conventional procedural programming languages, such as object-oriented programming languages ​​like Smalltalk, C++, and the "C" programming language, or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the present invention, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by personalizing the electronic circuitry using state information of the computer readable program instructions.

[0054] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams that illustrate methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0055] These computer-readable program instructions can be supplied to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to implement a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, form means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0056] The computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to realize a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0057] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially in parallel, or the blocks may possibly be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may embody a combination of dedicated hardware and computer instructions.

[0058] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0059] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0060] The features are as follows:

[0061] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without the need for human interaction with the service provider.

[0062] Wide network access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., cell phones, laptops, and PDAs).

[0063] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated depending on demand. Consumers generally have no control or knowledge of the exact location of the resources provided, but there is a sense of location independence in that they may be able to specify a higher level of abstraction (e.g., country, state, or data center).

[0064] Rapid Elasticity: Rapid elasticity allows you to quickly scale out, sometimes automatically, by provisioning capabilities, and quickly release capabilities to quickly scale in. To the consumer, the capabilities available for provisioning often appear infinite, and you can buy as many as you want, whenever you want.

[0065] Metered Services: Cloud systems automatically control and optimize resource utilization by utilizing metering capabilities at some level of abstraction appropriate for the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, giving both providers and consumers transparency into the services utilized.

[0066] The service model is as follows:

[0067] Software as a Service (SaaS): The functionality offered to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0068] Platform as a Service (PaaS): The capability offered to consumers is the deployment of consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, on a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application-hosting environment configuration.

[0069] Infrastructure as a Service (IaaS): The capability offered to consumers is the provisioning of processing, storage, networking, and other basic computing resources onto which they can deploy and run any software, which may include operating systems and applications. While consumers do not manage or control the underlying cloud infrastructure, they do have control over the operating systems, storage, and deployed applications, and may have limited control over selected networking components (e.g., host firewalls).

[0070] The deployment model is as follows:

[0071] Private Cloud: This cloud infrastructure is operated solely for the organization. It can be managed by the organization or a third party and can reside on-premise or off-premise.

[0072] Community Cloud: This cloud infrastructure is shared by several organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance concerns). It can be managed by the organization or a third party and can reside on-premises or off-premises.

[0073] Public Cloud: This cloud infrastructure is available to the public or large industry organizations and is owned by an organization that sells cloud services.

[0074] Hybrid Cloud: This cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain their own entities but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0075] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0076] FIG. 7 is a block diagram illustrating a cloud computing environment 50 in accordance with at least one embodiment of the present invention. The cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, can communicate. The nodes 10 can communicate with each other. The nodes 10 may be physically or virtually grouped (not shown) in one or more networks, such as the private cloud, community cloud, public cloud, or hybrid cloud described above. This enables the cloud computing environment 50 to provide infrastructure, platform, or software, or a combination thereof, as a service for which the cloud consumer does not need to maintain resources on their local computing device. It should be noted that the types of computing devices 54A-54N shown in FIG. 6 are intended to be illustrative only, and it is understood that computing node 10 and cloud computing environment 50 may communicate with any type of computerized device via any type of network connection or network-addressable connection, or combination thereof, (e.g., using a web browser).

[0077] Figure 8 is a block diagram illustrating a set of functional abstraction model layers provided by the cloud computing environment 50 shown in Figure 6, in accordance with at least one embodiment of the present invention. It should be understood in advance that the components, layers, and functions shown in Figure 8 are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0078] Hardware and software layer 60 includes hardware and software components, such as mainframe 61, reduced instruction set computer (RISC) architecture-based servers 62, servers 63, blade servers 64, storage devices 65, and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0079] The virtualization layer 70 provides an abstraction layer that can instantiate virtual entities such as: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0080] In one embodiment, management layer 80 may provide the following functionality: Resource provisioning 81 dynamically procures computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 tracks costs as resources are utilized within the cloud computing environment and bills or invoices for the consumption of those resources. In one embodiment, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks and protection of data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 allocates and manages cloud computing resources to ensure required service levels are met. Service level agreement (SLA) planning and fulfillment 85 proactively arranges for and procures cloud computing resources for anticipated future demand in accordance with SLAs.

[0081] The workload tier 90 provides examples of functions that can utilize a cloud computing environment. Examples of workloads and functions that can be delivered from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and speech recognition transcription correction 96.

[0082] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the present invention. The terms used herein have been selected to best explain the principles of the embodiments, practical applications, or technical improvements of the technology found in the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. 1. A computer-implemented method for improving speech recognition by processing information in a computer, comprising: one or more processors: Receiving a history of utterances and corresponding speech metrics of the utterances, wherein the history of utterances is associated with a particular node or a particular context for a prompt or question given by a virtual assistant; generating at least one utterance similar sound list for the particular node or particular context based on the utterance history and the phonetic metrics of the utterance; obtaining a transcription of the target utterance and a confidence level for the transcription for the particular node or particular context; in response to the confidence of the transcription being less than a threshold, modifying the transcription for which the confidence is less than the threshold based on a comparison of phonemes in the transcription with phonemes in the similar sound list for the particular node or particular context; A computer-implemented method for performing

2. 2. The computer-implemented method of claim 1, wherein the speech metrics identify a frequency of an utterance and a frequency of one or more terms corresponding to the utterance, or include a signal-to-noise ratio, background noise, speech ratio, high frequency loss, DC offset, clipping ratio, speech level, non-speech level, or a combination thereof.

3. 3. The computer-implemented method of claim 1, wherein obtaining a transcription of the target utterance and a confidence measure for the transcription comprises transcribing the target utterance with a deep neural network-based speech recognition model.

4. The similar sound list includes similar sounding phonemes for each phoneme in a high frequency term, and the modifying includes: calculating a score for a matched phoneme based on a comparison of the phoneme in the transcription with the phoneme in the high-frequency term and the phoneme of the similar sound list that sounds similar to the phoneme in the high-frequency term, and replacing the transcription whose confidence is below the threshold with a transcription corresponding to the matched phoneme if the score is equal to or greater than a threshold. The computer-implemented method of any one of claims 1 to 3, comprising:

5. The computer-implemented method of claim 4 , further comprising the one or more processors assigning similar sound values ​​to term phonemes that correspond to speech phonemes in the speech history.

6. The computer-implemented method of any one of claims 1 to 5, wherein the history of utterances is from the virtual assistant.

7. 6. The computer-implemented method of claim 5, further comprising the one or more processors storing the corresponding speech phonemes and term phonemes in a repository, the speech phonemes and term phonemes being the similar sound list.

8. 1. A computer system for improving speech recognition transcription, comprising: one or more computer processors; one or more computer-readable storage media; Including, Receiving a history of utterances and corresponding speech metrics of the utterances, wherein the history of utterances is associated with a particular node or a particular context for a prompt or question given by the virtual assistant; generating at least one utterance similar sound list for the particular node or particular context based on the utterance history and the phonetic metrics of the utterance; obtaining a transcription of the target utterance and a confidence level for the transcription for the particular node or particular context; and in response to the confidence of the transcription being below a threshold, modifying the transcription for which the confidence is below the threshold based on a comparison of phonemes in the transcription with phonemes in the similar sound list for the particular node or particular context; A computer system that performs operations including:

9. 9. The computer system of claim 8, wherein the speech metrics identify a frequency of an utterance and a frequency of one or more terms corresponding to the utterance, or include a signal-to-noise ratio, background noise, speech ratio, high frequency loss, DC offset, clipping ratio, speech level, non-speech level, or a combination thereof.

10. 10. The computer system of claim 9, wherein obtaining a transcription of the target utterance and a confidence measure for the transcription comprises transcribing the target utterance with a deep neural network based speech recognition model.

11. The similar sound list includes similar sounding phonemes for each phoneme in a high frequency term, and the modifying comprises: calculating a score for a matched phoneme based on a comparison of the phoneme in the transcription with the phoneme in the high-frequency term and the phoneme of the similar sound list that sounds similar to the phoneme in the high-frequency term, and replacing the transcription whose confidence is below the threshold with a transcription corresponding to the matched phoneme if the score is equal to or greater than a threshold. The computer system according to any one of claims 8 to 10, further comprising:

12. 12. The computer system of claim 11, further comprising instructions for assigning similar sound values ​​to vocabulary phonemes that correspond to speech phonemes in the speech history.

13. The computer system according to any one of claims 8 to 12, wherein the utterance history is from the virtual assistant.

14. 13. The computer system of claim 12, wherein the actions include storing the corresponding speech phonemes and vocabulary phonemes in a repository, the speech phonemes and vocabulary phonemes being a similar sound list.

15. A computer program for causing a computer to execute the method according to any one of claims 1 to 7.

16. A computer-readable recording medium having the computer program according to claim 15 recorded thereon.

Images