voice recognition

Abstract

One method (400) includes: for each corresponding audio segment (111) in a series of audio segments: generating multiple candidate speech recognition hypotheses (142) for the corresponding audio segment using a speech recognition model (150), and linking each corresponding candidate speech recognition hypothesis from the multiple candidate speech recognition hypotheses to a previously generated transcript (121) corresponding to N previous audio segments. Each corresponding candidate speech recognition hypothesis includes a corresponding probability (144). For each corresponding audio segment, the method further includes re-scoring the corresponding probability of each corresponding candidate speech recognition hypothesis based on the link (162) using a large language model (160), and generating a transcript (120) of the corresponding audio segment by selecting the corresponding candidate speech recognition hypothesis from the candidate speech recognition hypotheses that includes the highest re-scored probability (164).

Description

Technical Field

[0001] This disclosure relates to an automatic speech recognition (ASR) system that integrates non-autoregressive and multilingual language models. Background Technology

[0002] In recent years, Automatic Speech Recognition (ASR) systems have become increasingly popular for assistant-enabled devices. Improving word recognition performance remains a persistent challenge for ASR systems. This problem is further complicated by infrequently spoken words. That is, infrequently spoken words are rarely included in the training data, making it difficult for ASR systems to accurately recognize them in speech. In some cases, ASR systems incorporate language models trained on plain text data to improve the recognition of infrequently spoken words. However, using these language models often increases latency and requires significant memory and computational resources, making language model integration unsuitable for many applications. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the hardware to perform operations including: receiving a series of audio segments corresponding to speech spoken by a user; and for each corresponding audio segment in the series, generating a plurality of candidate speech recognition hypotheses for that corresponding audio segment using a speech recognition model; and linking each corresponding candidate speech recognition hypothesis from the plurality of candidate speech recognition hypotheses to a previously generated transcript corresponding to N previous audio segments. Each corresponding candidate speech recognition hypothesis includes a corresponding probability. For each corresponding audio segment in the series, the operation further includes re-scoring the corresponding probability of the corresponding candidate speech recognition hypothesis using a large language model (LLM) based on the link between each corresponding candidate speech recognition hypothesis and the previously generated transcript; and generating a transcript of the corresponding speech segment by selecting the corresponding candidate speech recognition hypothesis from the candidate speech recognition hypotheses that includes the highest re-scored probability.

[0004] Implementations of this disclosure may include one or more optional features from the following optional features. In some implementations, the speech recognition model includes an encoder and a decoder. Here, the encoder may generate higher-order feature representations for each corresponding audio segment by applying block-by-block bidirectional attention. Additionally, the encoder may include a stack of multi-head attention layers, each layer including a multi-head self-attention mechanism. Here, the stack of multi-head attention layers may include a stack of conformer layers. The stack of conformer layers may include a stack of 32 layers with approximately 2 billion parameters. Moreover, the decoder may include a connection-temporal classification (CTC) decoder. In these implementations, the decoder may non-autoregressively generate the plurality of candidate speech recognition hypotheses, and the LLM may non-autoregressively re-evaluate the corresponding probability of each corresponding candidate speech recognition hypothesis.

[0005] In some examples, the speech delivered by the user includes long utterances. In some additional examples, the N preceding audio segments immediately precede the corresponding audio segment.

[0006] Another aspect of this disclosure provides a system comprising data processing hardware and memory hardware storing instructions, which, when executed on the data processing hardware, cause the data processing hardware to perform operations including: receiving a series of audio segments corresponding to speech spoken by a user; and for each corresponding audio segment in the series, generating a plurality of candidate speech recognition hypotheses for that corresponding audio segment using a speech recognition model; and linking each corresponding candidate speech recognition hypothesis from the plurality of candidate speech recognition hypotheses to a previously generated transcript corresponding to N previous audio segments. Each corresponding candidate speech recognition hypothesis includes a corresponding probability. For each corresponding audio segment in the series, the operation further includes re-scoring the corresponding probability of the corresponding candidate speech recognition hypothesis using a large language model (LLM) based on the link between each corresponding candidate speech recognition hypothesis and the previously generated transcript; and generating a transcript of the corresponding speech segment by selecting the corresponding candidate speech recognition hypothesis from the candidate speech recognition hypotheses that includes the highest re-scored probability.

[0007] This aspect of the disclosure may include one or more optional features from the following optional features. In some implementations, the speech recognition model includes an encoder and a decoder. Here, the encoder may generate a higher-order feature representation for each corresponding audio segment by applying block-by-block bidirectional attention. Additionally, the encoder may include a stack of multi-head attention layers, each layer including a multi-head self-attention mechanism. Here, the stack of multi-head attention layers may include a stack of conformer layers. The stack of conformer layers may include a stack of 32 layers with approximately 2 billion parameters. Moreover, the decoder may include a connection-temporal classification (CTC) decoder. In these implementations, the decoder may non-autoregressively generate the plurality of candidate speech recognition hypotheses, and the LLM may non-autoregressively re-score the corresponding probability of each corresponding candidate speech recognition hypothesis.

[0008] In some examples, the speech delivered by the user includes long utterances. In some additional examples, the N preceding audio segments immediately precede the corresponding audio segment.

[0009] Details of one or more implementations of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of an example system for implementing a speech recognition system.

[0011] Figure 2 This is a schematic diagram of an example speech recognition system.

[0012] Figure 3 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0013] Figure 4 This is a flowchart illustrating an exemplary arrangement of operations for a method of re-scoring speech recognition results output from a non-autoregressive speech recognition model using non-autoregressive and multilingual language models.

[0014] In the various figures, the same reference numerals indicate the same elements. Detailed Implementation

[0015] Large-scale speech models, such as multilingual automatic speech recognition (ASR) models and multilingual large language models (LLMs), have recently shown significant performance improvements (e.g., word error rate (WER) and latency). Many speech applications (e.g., voice assistants and live commentary word addition) have strict latency constraints, making the integration of these large-scale ASR models and LLMs impractical. For example, the autoregressive nature of these models (e.g., generating each output sequentially based on the context of one or more previous input frames) leads to a significant increase in latency, making integration impractical for applications with strict latency constraints. On the other hand, the large model size of large-scale ASR models and LLMs makes their integration unsuitable for mobile applications with computational and / or memory constraints.

[0016] To this end, the implementation in this paper relates to a method and system for re-scoring speech recognition results output from a non-autoregressive speech recognition model using a non-autoregressive and multilingual language model. Specifically, the method includes receiving a series of audio segments corresponding to speech spoken by a user. For each corresponding audio segment, the method includes: generating multiple candidate speech recognition hypotheses, each including corresponding probabilities, using the speech recognition model; linking each corresponding candidate speech recognition hypothesis to a previously generated transcription corresponding to N previous audio segments; re-scoring the corresponding probabilities of each corresponding candidate speech recognition hypothesis based on the link using a large language model (LLM); and generating a transcription of the corresponding speech segment by selecting the corresponding candidate speech recognition hypothesis from the candidate speech recognition hypotheses that includes the highest re-scoring probability. Notably, the speech recognition model and the LLM operate non-autoregressively. That is, the speech recognition model and the LLM generate all outputs for each corresponding audio segment in parallel, rather than sequentially. Advantageously, by using the LLM to re-score multiple candidate speech recognition hypotheses, transcription accuracy is significantly increased, while operating with minimal latency due to the non-autoregressive operation of the speech recognition model and the LLM. Moreover, generating transcriptions for each audio segment that includes one or more acoustic frames is more efficient than generating transcriptions for each acoustic frame (i.e., shallow fusion).

[0017] Figure 1 A system 100 is illustrated, comprising a user 104 interacting with a user device 102 that communicates with and remotely computes device 201. System 100 includes an ASR model 150 implemented on the user device 102 and / or the remote computes device 201 (e.g., one or more servers of a distributed system executing in a cloud computing environment). Figure 2 ) and Large Language Model (LLM) 160 ( Figure 2The user device 102 is an automatic speech recognition (ASR) system 200. Although the user device 102 is depicted as a mobile computing device (e.g., a smartphone), the user device 102 can correspond to any type of computing device, such as, but not limited to, tablet devices, laptop / desktop computers, wearable devices, digital assistant devices, smart / speaker displays, smart home appliances, automotive infotainment systems, or Internet of Things (IoT) devices, and is equipped with data processing hardware and memory hardware 113.

[0018] User device 102 includes an audio subsystem 108 configured to receive spoken words from user 104 (e.g., user device 102 may include one or more microphones for recording spoken words 106) and convert the words 106 into a corresponding digital format associated with an input acoustic frame 110 (i.e., audio features) that can be processed by ASR system 200. In the example shown, user 104 speaks the phrase “What is the weather in New YorkCity?” in natural language English, and audio subsystem 108 converts the words 106 into corresponding acoustic frames 110 for input to ASR system 200. ASR model 150 then receives the acoustic frames 110 corresponding to the words 106 as input and generates / predicts a corresponding transcription 120 (e.g., recognition result / hypothesis) of the words 106 as output. In some examples, ASR system 200 segments the sequence of acoustic frames 110 into a series of audio segments 111 each comprising one or more acoustic frames 110. In the example shown, user device 102 and / or remote computing device 201 also execute a user interface generator 107 configured to present a representation of the transcription 120 of utterance 106 to user 104 of user device 102. In some configurations, the transcription 120 output from ASR system 200 is processed, for example, by LLM 160 with natural language understanding (NLU) capabilities to execute user commands. Alternatively or additionally, a text-to-speech system (e.g., executed on any combination of user device 102 or remote computing device 201) may convert the transcription 120 into synthesized speech for audible output to another device. For example, the original utterance 106 may correspond to a message that user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech for audible output to the friend so that the friend can hear the message conveyed in the original utterance.

[0019] refer to Figure 2The ASR system 200 includes an ASR model 150, an LLM 160, and an output layer 170. The ASR system 200 can operate non-autoregressively on a segment-by-segment basis, thereby generating a corresponding transcription 120 for each corresponding audio segment 111 in a series of audio segments 111. That is, instead of generating a corresponding transcription 120 for each acoustic frame 110 in the sequence of acoustic frames 110 (e.g., operating on a per-frame basis), the ASR system 200 processes the acoustic frames 110 from each audio segment 111 and generates a corresponding transcription 120 for each corresponding audio segment 111. Thus, the ASR system 200 can segment a sequence of acoustic frames 110 representing spoken utterances (e.g., speech spoken by one or more users) into a series of audio segments 111. Speech spoken by one or more users can include long-form utterances comprising minutes or hours of speech. For example, long-form utterances can include 15 minutes of speech input from a video for video captioning applications.

[0020] In some implementations, the ASR system 200 divides the sequence of acoustic frames 110 into one or more fixed-length audio segments 111, each comprising a fixed number of acoustic frames 110. For example, the ASR system 200 may divide a corresponding sequence of acoustic frames 110 comprising 30 seconds of audio into four (4) audio segments 111, each comprising eight (8) seconds of audio. In this example, the last audio segment 111 comprises six (6) seconds of audio and may be padded with an additional 2-second blank acoustic frame. Continuing with this example, the ASR system 200 processes each corresponding 8-second audio segment 111 to generate a corresponding transcription 120 for each corresponding audio segment 111. Moreover, the ASR system 200 operates in a streaming manner such that after processing each corresponding audio segment 111, the ASR system 200 outputs the corresponding transcription 120. That is, when the series of audio segments 111 each comprises 8 seconds of audio data, the ASR system 200 may output the corresponding transcription 120 every 8 seconds.

[0021] ASR model 150 may be a general speech model (USM) trained on multilingual training data including over 12 million hours of unlabeled audio, 28 billion sentences of text data, 110,000 hours of supervised audio data, and / or 100,000 hours of semi-supervised audio data. ASR model 150 includes an encoder 130 and a decoder 140. In some examples, encoder 130 has a stack of multi-head attention layers. For example, the stack of multi-head attention layers (e.g., 32 layers) may include a stack of conformer layers or transformer layers. Encoder 130 is configured to receive a series of audio segments 111 as input and generate, at each of a plurality of output steps, a higher-order feature representation 132 for a corresponding audio segment 111 from the series of audio segments 111. The higher-order feature representation 132 includes a series encoding representing an acoustic frame 110 from the audio segment 111. In some implementations, encoder 130 generates the higher-order feature representation 132 by applying chunk-wise bidirectional attention. Specifically, encoder 130 performs per-block bidirectional attention by performing attention on each audio segment 111 (i.e., block). Per-block bidirectional attention is more flexible than block processing, which forces all encoder layers to process context frames associated with the current block, by allowing other layers of encoder 130 to process context frames outside the current block.

[0022] The decoder 140 of the ASR model 150 is configured to receive a higher-order feature representation 132 generated by the encoder 130 at each output step as input, and to generate a plurality of candidate speech recognition hypotheses 142 for the corresponding higher-order feature representation 132 at each of the plurality of output steps. Each corresponding candidate speech recognition hypothesis 142 includes a corresponding probability 144 indicating the likelihood that the corresponding candidate speech recognition hypothesis 142 is an accurate transcription of speech from the corresponding audio segment 111. Therefore, the plurality of candidate speech recognition hypotheses 142 output by the decoder 140 may include a probability distribution 144 over the plurality of candidate speech recognition hypotheses 142. The plurality of candidate speech recognition hypotheses 142 correspond to a set of output labels, each representing a symbol / character of a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, for example, one label for each of the 26 letters of the English alphabet, and one label for a space. Therefore, the probability distribution 144 may indicate the likelihood of the occurrence of each output label in a predetermined set of output labels. In some configurations, output labels are graphemes (e.g., individual characters, and possibly punctuation marks and other symbols), but the set of output labels is not limited to this. For example, in addition to or alternative to graphemes, the set of output labels may include word segments and / or entire words. Probability distribution 144 may include posterior probability values ​​for each output label among the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, probability distribution 144 may include 100 different probability values, one probability value for each output label. Probability distribution 144 can then be used by output layer 170 to select candidate orthographic elements (e.g., graphemes, word segments, and / or words) and assign scores to them during beam search to determine transcription 120.

[0023] In some examples, decoder 140 includes a connection-temporal classification (CTC) decoder that operates non-autoregressively. That is, each higher-order feature representation 132 generated by encoder 130 may include an encoded sequence such that each encoded representation in the encoded sequence comes from a corresponding acoustic frame 110 of the corresponding audio segment 111. Thus, decoder 140 operates in parallel (e.g., non-autoregressively) by simultaneously generating corresponding candidate speech recognition hypotheses 142 for each acoustic frame 110 of the corresponding audio segment 111. Because decoder 140 operates in parallel, decoder 140 generates corresponding candidate speech recognition hypotheses 142 for each acoustic frame 110 of the corresponding audio segment 111 independently of each other acoustic frame 110 of the corresponding audio segment 111. In other words, the generation of each candidate speech recognition hypothesis 142 does not depend on any other acoustic frame 110 of the corresponding audio segment 111.

[0024] For example, decoder 140 can process a corresponding higher-order feature representation 212 generated from an 8-second audio segment 111 having 8 acoustic frames 110 (e.g., 8 separate 1-second acoustic frames 110) to generate 8 corresponding probability distributions 144 on candidate speech recognition hypotheses 142. Here, each probability distribution 144 on the candidate speech recognition hypotheses 142 corresponds to the recognition result for the corresponding 1-second acoustic frame in the 1-second acoustic frames 110. Therefore, in this example, the multiple candidate speech recognition hypotheses 142 output by decoder 140 can include every possible path traversing each of the 8 corresponding probability distributions 144 on the candidate speech recognition hypotheses 142. All possible paths form a confusion network mesh that grows exponentially with the length of the audio segment 111, making it challenging for LLM to re-evaluate each of the candidate speech recognition hypotheses. To address this, ASR model 150 operates on a fixed-length audio segment (e.g., 8 seconds) to limit the complexity of re-evaluation. In some implementations, decoder 140 may sum the corresponding probabilities 144 for each candidate speech recognition hypothesis 142 for each possible path and output an N-best list of candidate speech recognition hypotheses 142 with the highest summed probabilities. Compared to the CTC decoder architecture, other speech recognition architectures such as recurrent neural network converters (RNN-T) process each frame sequentially (e.g., generating one output at a time), whereby the output for each frame depends on one or more previous frames, resulting in increased latency.

[0025] ASR system 200 employs LLM 160 to re-evaluate the corresponding probability 144 for each corresponding candidate speech recognition hypothesis 142, generating a corresponding re-evaluated probability 164. That is, LLM 160 does not generate any new candidate speech recognition hypotheses 142, but instead re-evaluates the corresponding probability 144 generated by ASR model 150. LLM 160 may include a transformer architecture. For example, in some implementations, LLM 160 includes a Pathway Language Model 2 (PaLM 2) with a lexicalized 256k word fragment vocabulary and a transformer input dimension of 1536. LLM 160 is trained on multilingual training data including web documents and books.

[0026] In some examples, ASR model 150 generates a corresponding probability 144 for each candidate speech recognition hypothesis 142 based on the acoustic similarity between the candidate speech recognition hypothesis 142 and the corresponding acoustic frame 110. On the other hand, LLM 160 generates a corresponding re-scoring probability 164 based on the linguistic information (i.e., semantic interpretation) of the candidate speech recognition hypothesis 142. That is, because ASR model 150 generates multiple candidate speech recognition hypotheses 142 non-autoregressively and without considering the context of previous or subsequent acoustic frames, some candidate speech recognition hypotheses among the candidate speech recognition hypotheses 142 may be acoustically similar to spoken speech but have a low probability of linguistic accuracy. For example, due to the non-autoregressive operation of ASR model 150, ASR model 150 may not be able to disambiguate between the candidate speech recognition hypotheses 142 for “I have for dogs” and “I have four dogs”. However, in this example, LLM 160 may be able to determine, based on semantic interpretation, that the candidate speech recognition hypothesis 142 for “I have four dogs” has a greater probability of being accurate.

[0027] To this end, LLM 160 is configured to receive each corresponding candidate speech recognition hypothesis 142 and corresponding probability 144 from ASR model 150, as well as previously generated transcripts 121 corresponding to N previous audio segments 111, as input. For example, LLM 160 may receive 16 candidate speech recognition hypotheses 142 and corresponding 16 probabilities 144 generated by ASR model 150 for a corresponding audio segment 111, and previously generated transcripts 121 corresponding to the two (e.g., N = 2) previous audio segments 111 immediately preceding that corresponding audio segment 111. LLM 160 concatenates each corresponding candidate speech recognition hypothesis 142 from the plurality of candidate speech recognition hypotheses 142 with the previously generated transcripts 121 to generate a concatenation 162 for each corresponding candidate speech recognition hypothesis 142. In particular, LLM 160 may prepend the previously generated transcripts 121 to each corresponding candidate speech recognition hypothesis 142 to generate a corresponding concatenation 162 for each corresponding candidate speech recognition hypothesis 142. Subsequently, LLM 160 re-scores the corresponding probability 144 of each corresponding candidate speech recognition hypothesis 142 based on the connection 162 between that candidate hypothesis 142 and the previously generated transcription 121, to generate a corresponding re-scored probability 164. LLM 160 can generate the corresponding re-scored probability 164 by non-autoregressively re-scoring the corresponding probability 144 of each corresponding candidate speech recognition hypothesis 142. In some implementations, the re-scored probability 164 represents a new probability independent of the corresponding probability 144 generated by ASR model 150.

[0028] In some implementations, output layer 170 is configured to generate transcription 120 for each audio segment 111 by selecting the corresponding candidate speech recognition hypothesis with the highest re-score probability 164 from among the candidate speech recognition hypotheses 142 as transcription 120. Output layer 170 may be independent of or integrated with ASR model 150. In other implementations, output layer 170 is configured to generate transcription 120 for each audio segment 111 by selecting the corresponding candidate speech recognition hypothesis from among the candidate speech recognition hypotheses 142 based on a combination of probabilities, the probabilities of which include the corresponding probability 144 generated by ASR model 150 and the re-score probability generated by LLM 160. For example, output layer 170 may determine the probability of the combination based on:

[0029] In equation 1, This represents the combination probability for a specific audio segment 111. This indicates that the corresponding probability 144 is generated by ASR model 150. This represents the probability of a re-rating generated by LLM 160, which is 164. The LLM rating weight. In some examples, the LLM rating weight is equal to 0.3; however, the rating weight can be any value.

[0030] As described above, the ASR system 200 operates in a streaming manner by generating a corresponding transcription 120 at each of multiple output steps. Here, each output step corresponds to a corresponding audio segment in audio segment 111. Thus, if audio segment 111 comprises 8 seconds of audio, the ASR system 200 generates a corresponding transcription 120 every 8 seconds. Advantageously, the ASR system 200 operates non-autoregressively (compared to autoregressive architectures such as RNN-T) to increase the parallelization of inference, thereby reducing latency. Moreover, because the CTC decoder does not retain decoder state, the ASR system 200 is more robust to premature segmentation (e.g., word truncation). Another advantage of the ASR system 200 is that it operates on a segment-by-segment basis (rather than on a frame-by-frame basis (i.e., shallow fusion)), allowing the ASR system 200 to propagate a number of terms equal to the number of LLM 160 multiplied by the number of hypotheses output by the ASR model 150 at each output step. In contrast, the per-frame score requires a propagation equal to the number of acoustic frames multiplied by the number of hypotheses output by the ASR model 150 at each output step. Therefore, compared to the per-segment score, the per-frame score requires an additional propagation equal to the number of acoustic frames divided by the number of lexical units in the LLM 160.

[0031] Figure 3This is a schematic diagram of an example computing device 300 that can be used to implement the systems and methods described in this document. The computing device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementations of the invention described and / or claimed in this document.

[0032] The computing device 300 includes a processor 310, a memory 320, a storage device 330, a high-speed interface / controller 340 connected to the memory 320 and a high-speed expansion port 350, and a low-speed interface / controller 360 connected to a low-speed bus 370 and a storage device 330. Each of the components 310, 320, 330, 340, 350, and 360 is interconnected using various buses and may be mounted on a common motherboard or otherwise suitably mounted. The processor 310 (e.g., the data processing hardware 113 of the user device 102 or the data processing hardware of the remote system 60) can process instructions for execution within the computing device 300, including instructions stored in the memory 320 or on the storage device 330, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 380 coupled to the high-speed interface 340. In other implementations, multiple processors and / or multiple buses, as well as multiple memories and various types of memory, may be used as appropriate. In addition, multiple computing devices 300 can be connected (e.g., as a server library, blade server group, or multiprocessor system), each of which provides some of the necessary operations.

[0033] Memory 320 (e.g., memory hardware 113 of user device 102 or memory hardware of remote system 60) stores information non-transitorily within computing device 300. Memory 320 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 320 may be a physical means for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 300. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0034] Storage device 330 provides mass storage for computing device 300. In some implementations, storage device 330 is a computer-readable medium. In various implementations, storage device 330 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array (including devices arranged in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 320, storage device 330, or memory on processor 310.

[0035] High-speed controller 340 manages bandwidth-intensive operations of computing device 300, while low-speed controller 360 manages lower bandwidth-intensive operations. This assignment of responsibilities is merely exemplary. In some implementations, high-speed controller 340 is coupled to memory 320, display 380 (e.g., via a graphics processor or accelerator), and high-speed expansion port 350 which can accept various expansion cards (not shown). In some implementations, low-speed controller 360 is coupled to storage device 330 and low-speed expansion port 390. Low-speed expansion port 390, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled, for example, via a network adapter to one or more input / output devices such as a keyboard, pointing device, scanner, or a networking device such as a switch or router.

[0036] The computing device 300 can be implemented in a variety of different forms, as shown in the figure. For example, the computing device can be implemented as a standard server 300a or multiple implementations in a group of such servers 300a, as a laptop computer 300b, or as part of a rack server system 300c.

[0037] Figure 4 This is a flowchart illustrating an exemplary arrangement of operations for a computer-implemented method 400 for re-scoring speech recognition results output from a non-autoregressive speech recognition model using non-autoregressive and multilingual language models. These operations can be performed by data processing hardware 310 ( Figure 3 (For example, the data processing hardware 11 of the user device 10 or the data processing hardware of the remote computing system 60) executes based on execution instructions stored on the memory hardware 320 (for example, the memory hardware 13 of the user device 10 or the memory hardware of the remote computing system) that communicates with the data processing hardware 310.

[0038] In operation 402, method 400 includes receiving a series of audio segments 111 corresponding to the speech spoken by user 104. Operations 404–410 are performed for each corresponding audio segment 111 in the series of audio segments 111.

[0039] In operation 404, method 400 includes generating a plurality of candidate speech recognition hypotheses 142 for corresponding audio segments 111 using speech recognition model 150. Here, each corresponding candidate speech recognition hypothesis 142 includes a corresponding probability 144. In operation 406, method 400 includes linking each corresponding candidate speech recognition hypothesis 142 from the plurality of candidate speech recognition hypotheses 142 with previously generated transcripts 121 corresponding to N previous audio segments 111.

[0040] In operation 408, method 400 includes re-scoring the corresponding probability 144 of each corresponding candidate speech recognition hypothesis 142 based on the link 162 between each corresponding candidate speech recognition hypothesis 142 and the previously generated transcript 121 using a large language model (LLM) 160. In operation 410, method 400 includes generating a transcript 120 of the corresponding speech segment 111 by selecting the corresponding candidate speech recognition hypothesis from the candidate speech recognition hypotheses 142 that includes the highest re-scored probability 164.

[0041] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose and is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0042] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0043] The processes and logic flows described in this specification can be executed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to said mass storage device, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry systems.

[0044] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) and possibly a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0045] Various implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (400) executed on data processing hardware (310) to cause said data processing hardware (310) to perform operations including: Receives a series of audio segments (111) corresponding to the speech spoken by the user; and For each corresponding audio segment (111) in the series of audio segments (111): A speech recognition model (150) is used to generate a plurality of candidate speech recognition hypotheses (142) for the corresponding audio segment (111), each corresponding candidate speech recognition hypothesis (142) including a corresponding probability (144); Each corresponding candidate speech recognition hypothesis (142) from the plurality of candidate speech recognition hypotheses (142) is linked with the previously generated transcripts (121) corresponding to N previous audio segments (111); The corresponding probabilities (144) of the respective candidate speech recognition hypotheses (142) are re-scored using a large language model (LLM) (160) based on the link (162) between each corresponding candidate speech recognition hypothesis (142) and the previously generated transcript (121); and Transcriptions of the corresponding speech segments are generated by selecting the corresponding candidate speech recognition hypothesis (164) from the candidate speech recognition hypotheses (142).

2. The computer-implemented method (400) as claimed in claim 1, wherein the speech recognition model (150) includes an encoder (130) and a decoder (140).

3. The computer-implemented method (400) of claim 2, wherein the encoder (130) generates a higher-order feature representation (132) for each corresponding audio segment (111) by applying block-by-block bidirectional attention.

4. The computer-implemented method (400) as claimed in claim 2 or 3, wherein the encoder (130) comprises a stack of multi-head attention layers, each including a multi-head self-attention mechanism.

5. The computer-implemented method (400) of claim 4, wherein the stacking of the multi-head attention layers includes the stacking of conformer layers.

6. The computer-implemented method (400) of claim 5, wherein the stack of conformer layers comprises a stack of 32 layers having approximately 2 billion parameters.

7. The computer-implemented method (400) of any one of claims 2–6, wherein the decoder (140) comprises a Connection Timing Classification (CTC) decoder.

8. The computer-implemented method (400) as described in any one of claims 2–7, wherein: The decoder (140) generates the plurality of candidate speech recognition hypotheses (142) in a non-autoregressive manner; as well as The LLM (160) non-autoregressively re-scored the corresponding probability (144) of each corresponding candidate speech recognition hypothesis (142).

9. The computer-implemented method (400) of any one of claims 1–8, wherein the speech spoken by the user comprises a long-form utterance (106).

10. The computer-implemented method (400) of any one of claims 1–9, wherein the N preceding audio segments (111) immediately precede the respective audio segment (111).

11. A system (100) comprising: Data processing hardware (310); as well as A memory hardware (320) communicating with the data processing hardware (310) stores instructions that, when executed on the data processing hardware (310), cause the data processing hardware (310) to perform operations including: Receives a series of audio segments (111) corresponding to the speech spoken by the user; and For each corresponding audio segment (111) in the series of audio segments (111): A speech recognition model (150) is used to generate a plurality of candidate speech recognition hypotheses (142) for the corresponding audio segment (111), each corresponding candidate speech recognition hypothesis (142) including a corresponding probability (144); Each corresponding candidate speech recognition hypothesis (142) from the plurality of candidate speech recognition hypotheses (142) is linked with the previously generated transcripts (121) corresponding to N previous audio segments (111); The corresponding probabilities (144) of the respective candidate speech recognition hypotheses (142) are re-scored using a large language model (LLM) (160) based on the link (162) between each corresponding candidate speech recognition hypothesis (142) and the previously generated transcript (121); and Transcriptions of the corresponding speech segments are generated by selecting the corresponding candidate speech recognition hypothesis (164) from the candidate speech recognition hypotheses (142).

12. The system (100) of claim 11, wherein the speech recognition model (150) includes an encoder (130) and a decoder (140).

13. The system (100) of claim 12, wherein the encoder (130) generates a higher-order feature representation (132) for each corresponding audio segment (111) by applying block-by-block bidirectional attention.

14. The system (100) of claim 12 or 13, wherein the encoder (130) comprises a stack of multi-head attention layers, each including a multi-head self-attention mechanism.

15. The system (100) of claim 14, wherein the stacking of the multi-head attention layers includes the stacking of conformer layers.

16. The system (100) of claim 15, wherein the stack of the conformer layers comprises a stack of 32 layers having approximately 2 billion parameters.

17. The system (100) of any one of claims 12–16, wherein the decoder (140) comprises a connection timing classification (CTC) decoder (140).

18. The system (100) as claimed in any one of claims 12–17, wherein: The decoder (140) generates the plurality of candidate speech recognition hypotheses (142) non-autoregressively; and The LLM (160) non-autoregressively re-scored the corresponding probability (144) of each corresponding candidate speech recognition hypothesis (142).

19. The system (100) of any one of claims 11–18, wherein the speech spoken by the user comprises a long-form utterance (106).

20. The system (100) of any one of claims 11–19, wherein the N preceding audio segments (111) immediately precede the respective audio segment (111).

Images