Combined Segmentation and Automatic Speech Recognition

Abstract

The combined segmentation and ASR model (200) includes an encoder (220) and a decoder (260). The encoder is configured to receive acoustic frames (110) characterizing one or more utterances (106) and generate, at each output step, a high-level feature representation (224) of the corresponding acoustic frame. The decoder is configured to receive the high-level feature representation (224) and generate, at each output step, a probability distribution (242) over possible speech recognition hypotheses and an indication (212) of whether the corresponding output step corresponds to the end of a speech segment. The combined segmentation and ASR model, trained on a set of training samples, each includes audio data characterizing an uttered speech and a corresponding transcription of the uttered speech, where the corresponding transcription automatically inserts a speech segment end ground truth token into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training samples.

Description

[Technical field]

[0001] The present disclosure relates to speech segmentation and automatic speech recognition. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems are focused on delivering not only good quality (e.g., low word error rate) but also low latency (e.g., a short delay between when a user speaks and when a transcription appears). For example, when using a device that implements an ASR system today, one often expects the ASR system to decode speech in a streaming format that responds in real time or even faster than real time. Summary of the Invention

[0003] One aspect of the present disclosure provides a combined segmentation and automatic speech recognition (ASR) model including an encoder and a decoder. The encoder is configured to receive as input a sequence of acoustic frames characterizing one or more utterances and generate, at each of a plurality of output steps, a high-order feature representation of a corresponding acoustic frame of the sequence of acoustic frames. The decoder is configured to receive as input the high-order feature representation generated by the encoder at each of the plurality of output steps and generate, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses and an indication of whether the corresponding output step corresponds to an end of a speech segment. The combined segmentation and ASR model is trained with a set of training samples, where each training sample of the set of training samples includes audio data characterizing an uttered speech and a corresponding transcription of the uttered speech, where a speech segment end ground truth token is automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training sample.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the decoder includes a prediction network, a first combination network, and a second combination network. The prediction network is configured to receive as input a sequence of non-blank symbols output by the final softmax layer at each of the multiple output steps and generate a hidden representation. The first combination network is configured to receive as input a hidden representation generated by the prediction network at each of the multiple output steps and a high-order feature representation generated by the encoder at each of the multiple output steps and generate an indication at each of the multiple output steps of whether the corresponding output step corresponds to an end of a speech segment. The second combination network is configured to receive as input a hidden representation generated by the prediction network at each of the multiple output steps and a high-order feature representation generated by the encoder at each of the multiple output steps and generate a probability distribution over possible speech recognition hypotheses at each of the multiple output steps.

[0005] In some examples, in each of the multiple output steps, the sequence of previous non-blank symbols received as input at the prediction network includes a sequence of N previous non-blank symbols output by a final softmax layer, where the prediction network is configured to generate a hidden representation by generating a respective embedding for each non-blank symbol of the sequence of N previous non-blank symbols, and generating an average embedding by averaging the respective embeddings, where the average embedding includes the hidden representation. In some examples, the prediction network includes a V2 embedding lookup table. In some implementations, the training process trains a combined segmentation and ASR model on the set of training samples by, during a first stage, training a second combination network to learn how to predict a corresponding transcription of an utterance characterized by the audio data of each training sample, and after training the second combination network, during a second stage, initializing the first combination network with the same parameters as the trained second combination network, and using the audio segment termination ground truth tokens inserted into the corresponding transcription of an utterance characterized by the audio data of each training sample.

[0006] In some implementations, the encoder comprises a causal encoder that includes a stack of conformer or transformer layers. In some examples, the audio segment end ground truth tokens are automatically inserted into the corresponding transcription without any human annotation.

[0007] In some examples, the set of heuristic-based rules and exceptions applied to each training sample of the set of training samples includes inserting a ground truth speech segment end token at the end of the corresponding transcription, and inserting the ground truth speech segment end token into the corresponding transcription at a position aligned with a non-speech segment of the audio data having a duration that meets a threshold duration unless the non-speech segment of the audio data follows a word of the uttered speech that is identified as a lengthened word or the non-speech segment of the audio data follows a word of the uttered speech that is identified as a filler word. In some implementations, a word of the uttered speech is identified as a lengthened word when a phoneme duration of the word meets a standard deviation threshold. In some examples, after training the combined segmentation and ASR model, the decoder is configured to emit an indication that the corresponding output step corresponds to the end of the speech segment sooner than identifying a number of consecutive non-speech acoustic frames of the sequence of acoustic frames that meet the threshold duration. In some implementations, the combined segmentation and ASR model is trained to maximize the probability of emitting a speech segment end ground truth label.

[0008] Another aspect of the disclosure provides a computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations, the operations including receiving as input a sequence of acoustic frames characterizing one or more utterances, and generating, at each of a plurality of output steps, by an encoder of the combined segmentation and automatic speech recognition (ASR) model, a high-order feature representation of a corresponding acoustic frame of the sequence of acoustic frames, and by a decoder of the combined segmentation and ASR model, a probability distribution over possible speech recognition hypotheses and an indication of whether the corresponding output step corresponds to an end of a speech segment. The combined segmentation and ASR model is trained with a set of training samples, where each training sample of the set of training samples includes audio data characterizing an uttered speech and a corresponding transcription of the uttered speech, where a speech segment end ground truth token is automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training sample.

[0009] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the operations further include using a predictive network of the decoder to generate a hidden representation based on the sequence of non-blank symbols output by the final softmax layer at each of the multiple output steps, generating an indication of whether the corresponding output step corresponds to an end of the speech segment includes using a first combined network of the decoder to generate an indication of whether the corresponding output step corresponds to an end of the speech segment based on the hidden representation generated by the predictive network at each of the multiple output steps and the high-order feature representation generated by the encoder at each of the multiple output steps, and generating a probability distribution over the possible speech recognition hypotheses includes using a second combined network of the decoder to generate a probability distribution over the possible speech recognition hypotheses at each of the multiple output steps based on the hidden representation generated by the predictive network and the high-order feature representation generated by the encoder at each of the multiple output steps.

[0010] In some examples, the sequence of previous non-blank symbols received as input at the prediction network includes a sequence of N previous non-blank symbols output by a final softmax layer, and generating the hidden representation using the prediction network includes generating the hidden representation by: generating a respective embedding for each non-blank symbol of the sequence of N previous non-blank symbols, and generating an average embedding by averaging the respective embeddings, where the average embedding includes the hidden representation. In some implementations, the prediction network includes a V2 embedding lookup table. In some examples, the training process trains the combined segmentation and ASR model on the set of training samples by, during a first stage, training a second combination network to learn how to predict a corresponding transcription of an utterance characterized by the audio data of each training sample, and after training the second combination network, during a second stage, initializing the first combination network with the same parameters as the trained second combination network, and using the audio segment termination ground truth tokens inserted into the corresponding transcription of an utterance characterized by the audio data of each training sample.

[0011] In some implementations, the encoder comprises a causal encoder including a stack of conformer or transformer layers. In some examples, the speech segment end ground truth token is automatically inserted into the corresponding transcription without any human annotation. In some implementations, a set of heuristic-based rules and exceptions applied to each training sample of the set of training samples includes inserting a ground truth speech segment end token at the end of the corresponding transcription, and inserting the ground truth speech segment end token into the corresponding transcription at a location aligned with a non-speech segment of the audio data having a duration that meets a threshold duration unless the non-speech segment of the audio data follows a word of the utterance that is identified as a lengthened word or the non-speech segment of the audio data follows a word of the utterance that is identified as a filler word. In some examples, a word of the utterance is identified as a lengthened word when the phoneme duration of the word meets a standard deviation threshold. In some embodiments, after training the joint segmentation and ASR model, the operations also include emitting, by the decoder, an indication that the corresponding output step corresponds to the end of a speech segment sooner than identifying a number of consecutive non-speech audio frames in the sequence of audio frames that satisfy the threshold duration.

[0012] In some examples, a joint segmentation and ASR model is trained to maximize the probability of emitting an audio segment end ground truth label.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0014] [Figure 1]FIG. 1 is a schematic diagram of an exemplary audio environment using an automatic speech recognition (ASR) system including a speech segment termination combination network for transcribing spoken speech. [Diagram 2] FIG. 1 is a schematic diagram of an exemplary ASR system incorporating an audio segment termination combining network. [Diagram 3] FIG. 3 is a schematic diagram of an example concatenation and reduction prediction network of the ASR system of FIG. 2. [Figure 4] 1 is a flowchart of an exemplary arrangement of operations of a computer-implemented method of implementing an ASR system including a speech segment termination model. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] Improving the performance of end-to-end (E2E) ASR models for long utterances ranging in length from a few minutes to a few hours is a persistent challenge in speech recognition. Streaming E2E ASR models have achieved low word error rates (WER) for short-to-medium length utterances up to a few minutes in length. However, traditional E2E ASR models can have high WER and suffer from deletion errors for long-form utterances ranging in length from tens of minutes to a few hours. Such utterances are found, for example, in tasks such as transcription of meetings and lectures, and video captioning.

[0017] Traditionally, long-form speech is first segmented using a separate voice activity detector (VAD). Whenever the VAD detects a long silence, it splits the audio at that location into two segments, which are then processed separately or independently by the E2E ASR model. At each segment boundary, the beam search finalizes the top hypothesis by discarding all other hypotheses. This introduces more diversity into the beam search by occasionally wiping out old hypotheses and making space for new ones, ultimately improving the WER by confirming more potentially correct hypotheses. Maintaining beam diversity is especially important for E2E ASR models, which are typically decoded with small beams. However, VAD-based segmentators suffer from high latency because, by design, the VAD must wait on long silences before making a segmentation decision. This delays subsequent ASR functions such as rescoring or prefetching, which must wait for the top hypothesis to finish. Improving latency is important because it can improve the user experience by improving the responsiveness of smart assistants through faster prefetching, or by helping dictation or captioning apps reduce the amount of "flickering" caused by switching between top hypotheses. Current VAD-based segmentators also suffer from high segmentation errors because VAD bases its decisions purely on audio rather than decoded text, which may contain semantic cues about when to segment. Improving segmentation accuracy is important because it can improve the WER. On the other hand, not segmenting at all leads to an inflated beam search with no diversity in hypotheses, which may also induce word errors.

[0018] The embodiments herein are directed toward integrating an ASR system (e.g., including a Recurrent Neural Network-Transducer (RNN-T) model) with a speech segment termination model (e.g., an additional combination network) that is configured and trained to predict whether a corresponding output step of the ASR system corresponds to the end of a speech segment event based on acoustic and semantic features. The disclosed ASR system is jointly optimized for both the detection of speech segment termination and the ASR task. However, existing long utterance training data for ASR systems usually does not include ground truth labels corresponding to speech segment termination, making supervised training of speech segment termination combinations difficult. The embodiments herein are directed toward performing a speech segment termination annotation routine for automatically inserting speech segment termination ground truth labels into ground truth training transcripts based on modeling hesitation and word timing. An exemplary speech segment termination scheme applies a heuristic-based weakly supervised approach, whereby speech segment termination ground truth labels are automatically inserted into ground truth training transcripts by applying one or more rules.

[0019] FIG. 1 is a schematic diagram of an example system 100 and an audio environment 102. In the audio environment 102, a way in which a user 104 interacts with a computing device, such as a user device 10, can be through voice input. The user device 10 (also commonly referred to as device 10) is configured to capture sounds (e.g., streaming audio data) from one or more users 104 within the audio environment 102. Here, the streaming audio data can refer to voiced utterances 106 by the users 104 that function as audible queries, commands to the device 10, or audible communications captured by the device 10. The voice-enabled system of the device 10 can address the queries or commands by responding to the queries and / or causing the commands to be executed / accomplished by one or more downstream applications. The voice-enabled system of the device 10 can also generate transcriptions of the audible communications captured by the device 10.

[0020] The user device 10 may correspond to any computing device associated with a user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, in-vehicle infotainment systems, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12. The memory hardware 14 stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 with audio capture devices (e.g., microphones) 16, 16a for capturing and converting uttered speech 106 into electrical signals within the audio environment 102, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the user device 10). In the illustrated example, the user device 10 implements a single audio capture device 16a, but the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, such that one or more capture devices 16a of the array may communicate with the audio system 16 without being physically present at the user device 10.

[0021] In the system 100, the automatic speech recognition (ASR) system 118 includes an RNN-T model 200 that implements a speech segment termination model 205 and resides on a user device 10 of a user 104 and / or a remote computing device 60 (e.g., one or more remote servers of a distributed system running in a cloud computing environment) that communicates with the user device 10 via a network 40. The user device 10 and / or the remote computing device 60 also includes an audio subsystem 108 that is configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16a and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 118. In the illustrated example, the user speaks each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., acoustic frames) 110 for input to the ASR system 118. The RNN-T model 200 then receives as input an acoustic frame 110 corresponding to the utterance 106, generates an indication of whether a corresponding output step corresponds to an end of a speech segment that can be used to split the utterance 106 into speech segments, and generates / predicts as output a transcription 120 corresponding to each speech segment of the utterance 106. In the illustrated example, the RNN-T model 200 can perform streaming speech recognition to generate partial transcriptions (e.g., streaming speech recognition results) 120a, 120b as the user speaks.

[0022] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120, 120a-n of the utterance 106 to the user 104 of the user device 10 and to present one or more responses 121, 121a-n to the queries and / or commands of the utterance 106. As described in more detail below, the user interface generator 107 may display the speech recognition results 120, 120a-n and the responses 121, 121a-n over time or at different times. As shown, the user interface generator 107 may present the speech recognition results 120, 120a-n (e.g., the transcription) and the responses 121, 121a-n to represent an interaction / conversation between the user 104 and an interactive program or application (e.g., the digital assistant application 50).

[0023] In some configurations, the transcription 120 output from the ASR system 118 for each identified speech segment is processed, for example, by a natural language processing / understanding (NLP / NLU) module running on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. The digital assistant application 50 may provide an appropriate response 121 after executing the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., running on any combination of the user device 10 or the remote computing device 60) may convert the transcription 120 and / or the response 121 into synthetic speech for audible output by the user device 10 and / or another device.

[0024] In the illustrated example, a user 104 interacts with a program or application (e.g., a digital assistant application 50) of the user device 10 using the ASR system 118. For example, FIG. 1 shows a user 104 communicating with the digital assistant application 50 and the digital assistant application 50 displayed in an interactive digital assistant interface 18 on a screen 19 of the user device 10 to illustrate a natural conversational interaction between the user 104 and the digital assistant application 50. In this example, the user 104 utters a first speech segment 106a of an utterance 106 ("Dictate my next message") and then a second speech segment 106b of the utterance 106 ("Where can I order Thai food?").

[0025] When the ASR system 118 receives an acoustic frame 110 corresponding to an utterance 106 as the user 104 utters it, the speech segment termination model 205 generates, at each output step, an indication of whether the output step corresponds to the end of a speech segment. Using the speech segment termination indications generated by the speech segment termination model 205, the speech segments 106 of the utterance 106 may be delineated, and the RNN-T model 200 may perform speech recognition on each speech segment 106 individually. In the illustrated example, the speech segment termination model 205 detects or determines a speech segment termination between a first speech segment 106 and a second speech segment 106a, and outputs a speech segment termination indication indicating the end of the first speech segment 106a.

[0026] For each speech segment 106 of the utterance 106 indicated or identified by the speech segment termination model 205, the RNN-T model 200 performs speech recognition on the acoustic frames 110 corresponding to the speech segment 106 to generate a speech recognition result 120 corresponding to the speech segment 106. Continuing with this example, based on the speech segment termination model 205 detecting an end segment of speech between the first speech segment 106 and the second speech segment 106, at time 1, the user interface generator 107 presents a display of the first speech recognition result 120a for the first speech segment 106a to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming format, whereby words, word pieces, and / or individual characters are displayed on the screen 19 of the user device 10 as soon as they are uttered.

[0027] Continuing with this example, based on the speech segment end model 205 detecting the end segment of speech for the second speech segment 106, at time 2, the user interface generator 107 presents a display of the second speech recognition result 120b for the second speech segment 106b to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming format, whereby words, word pieces, and / or individual characters are displayed on the screen 19 of the user device 10 as soon as they are uttered. Then, during time 3, the user interface generator 107 presents, via the digital assistant interface 18, a response 121b (i.e., "Send a message?") generated by the assistant application 50 based on the speech recognition results 120a, 120b that have been generated for the speech segments 106a, 106b.

[0028] 2 is a schematic diagram of an exemplary RNN-T model 200 that integrates a speech segment termination model 205 as a first combination network 210 for detecting boundaries between uttered speech segments. As shown, the RNN-T model 200 includes an encoder network 220, a prediction network 300, a second combination network 240 (i.e., wordpiece combination network 240), and a final softmax output layer 250. The encoder network 220 (e.g., an audio encoder), which is roughly similar to an acoustic model (AM) in a conventional ASR system, generates a sequence of feature vectors.

number

number

number

[0029] In the illustrated example, the prediction network 300 includes an LSTM-based prediction network, which, like a language model (LM), predicts the sequence of non-blank symbols previously output by the softmax layer 250.

number

number

number

[0030] Figure 3 shows the input, y u-n ,...,y u-1 1 illustrates a prediction network 300 for an RNN-T model 200 that receives a sequence of N non-blank symbols 252a-n, which is limited to the N previous non-blank symbols 252a-n output by the final softmax layer 250. In some examples, N is equal to 2. In other examples, N is equal to 5, but this disclosure is non-limiting and N may be equal to any integer. The sequence of non-blank symbols 252a-n may represent a partial speech recognition result 120a, 120b (FIG. 1). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302, which shares a shared embedding matrix 304 across each head 302A-H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads may be used by the multi-head attention mechanism 302. Notably, the multi-head attention mechanism 302 significantly improves performance with a minimal increase in the size of the model. As explained in more detail below, each head 302A-H contains its own row of position vectors 308, and rather than increasing the model size by concatenating the outputs 318A-H from all heads, the outputs 318A-H are instead averaged by a head averaging module 322.

[0031] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses a shared embedding matrix 304 to estimate the sequence of non-empty symbols y received as input at a corresponding output step from multiple output steps. ui-n , ..., y ui-1 For each non-whitespace symbol 301 in

number

number

[0032] A sequence of non-blank symbols 252a-n, y un,..., y The corresponding embedding generated by the shared embedding matrix 304 for each non-blank symbol 301 in u-1 is the same for all of the heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines a position vector PV Aa~An The second head 302B defines a position vector PV Ba~Bn 308 of Ba~Bn , ..., the Hth head 302H has a position vector PV Ha~Hn 308 of Ha~Hn Define another different line of

[0033] For each non-blank symbol in the received sequence of non-blank symbols 252a-n, the first head 302A, via the weight layer 310, weights the corresponding embedding 306 in proportion to the similarity between the corresponding embedding and the respective position vector PV 308 assigned to it. In some examples, the similarity may include a cosine similarity (e.g., cosine distance). In the example shown, the weight layer 310 outputs a sequence of weighted embeddings 312, 312Aa-An, each of which is weighted corresponding embedding 306 in proportion to the respective position vector PV 308 assigned to it. Stated differently, the weighted embedding 312 output by the weight layer 310 for each embedding 306 may correspond to a dot product between the embedding 306 and the respective position vector PV 308. The weighted embedding 312 may be interpreted as overriding the embedding in proportion to how similar the embedding is to the positioning associated with the respective position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, and therefore the sequence of weighted embeddings 312Aa-An are not concatenated but instead averaged by a weighted average module 316 to produce a weighted average 318A of the weighted embeddings 312Aa-An as output from the first head 302A, expressed as follows:

number

[0034] The operations described above with respect to the first head 302A are similarly performed for each of the other heads 302B-H of the multi-head attention mechanism 302. Due to the different sets of positioned vectors PV 308 defined by each head 302, the weight layer 310 outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn for each of the other heads 302B-H that differs from the sequence of weighted embeddings 312Aa-Aa for the first head 302A. The weighted average module 316 then generates a weighted average 318B-H of each of the corresponding weighted embeddings 312 of the sequence of non-blank symbols as output from the corresponding heads 302B-H.

[0035] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages weighted averages 318A-H output from corresponding heads 302A-H. A projection layer 326 with SWISH may receive as input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H and generate as output a projection output 328. A final layer normalization 330 normalizes the projection output 328 to produce a single embedding vector at the corresponding output step from the multiple output steps.

number

number

[0036] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302 and only performs the operations described above for the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer 326 and layer normalization 330 to produce a single embedding vector

number

[0037] Referring back to FIG. 2A, the speech segment termination combination network 210 (i.e., the first combination network 210) extracts a single embedding vector (V) from the prediction network 300.

number

number

number

number

number

number

[0038] In some implementations, the training data generator 140 (see FIG. 1) uses a heuristic-based, weakly supervised approach to generate ground truth <eos>Automatically insert tokens into paired audio-transcription training data (e.g., without any human annotation) to obtain ground truth <eos>Tokens are automatically inserted based on one or more rules. Exemplary rules include, but are not limited to, matching ground truth audio segment ends with the ends of the corresponding transcriptions. <eos>Insert tokens and ground truth audio segment ends <eos>The method includes inserting a token into the corresponding transcription at a position aligned with a non-speech segment of the audio data (e.g., silence) having a duration that meets the threshold duration unless the non-speech segment of the audio data follows a word of the spoken speech identified as a lengthened word (e.g., "heyyy") or the non-speech segment of the audio data follows a word of the spoken speech identified as a filler word (e.g., "um"). Such exceptions reflect situations where the speaker is likely not finished speaking a speech segment. A lengthened word may be detected, for example, when a phoneme duration meets a standard deviation threshold (e.g., when the phoneme duration is more than five times the standard deviation of the phoneme duration). Filler words may be detected using a filler word detection model. In some implementations, the training data generator 140 applies the rules and exceptions by using a forced alignment model that is applied to all audio transcription pairs in the training data set to obtain the timing of silence, words, and phonemes.

[0039] The wordpiece combination network 240 (i.e., the second combination network 240) extracts a single embedding vector from the prediction network 300.

number

number

number

number

number

number

[0040] A speech segment end combining network 210 detects the end of a speech segment (e.g., <eos>Token) 212, the softmax layer 250 calculates the next output symbol predicted by the RNN-T model 200 at the corresponding output step.

number

number

[0041] In some implementations, the feature vectors input to the encoder network 220 are

number

[0042] The prediction network 300 may include a LTSM-based network with an embedding dimension of 320.

number

number

[0043] The speech segment termination combination network 210, the wordpiece combination network 240, and the prediction network 300 may collectively form the RNN-T decoder 260 of the RNN-T model 200. In some implementations, to further reduce the size of the RNN-T decoder 260, i.e., the prediction network 300 and the wordpiece combination network 240, a parameter combination between the prediction network 300 and the wordpiece combination network 240 is applied. In particular, the vocabulary size |V| and the embedding dimension d e Then the shared embedding matrix 304 in the prediction network is

number

number

[0044] In some implementations, the RNN-T model 200 performs a frame-synchronous beam search with a beam size of 8 and a pruning threshold of 5 (i.e., negative hypotheses whose negative log posterior exceeds that of the top hypothesis by 5 are removed). At each frame, the RNN-T model 200 performs a breadth-first search for possible expansions, ignoring any expansions with a negative log posterior of 5 or more, and limiting the search depth to 10 expansions. In some examples, the RNN-T model 200 enforces the termination of speech segments that exceed a duration of 65 seconds.

[0045] 4 is a flowchart of an exemplary operational arrangement of a computer-implemented method 400 for performing joint segmentation and automatic speech recognition. At operation 402, the method 400 generates a sequence of acoustic frames (e.g.,

number

[0046] For each of the plurality of output steps, the method 400 performs operations 404, 406, and 408. In operation 404, the method 400 generates, for each of the plurality of output steps, a high-level feature representation of a corresponding acoustic frame of the sequence of acoustic frames by the encoder network 220 of a joint segmentation and automatic speech recognition (ASR) model (e.g., RNN-T200).

number

[0047] In operation 406, the method 400 generates a probability distribution over the possible speech recognition hypotheses by the decoder 260 of the joint segmentation and ASR model.

number

[0048] 5 is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described herein. The computing device 500 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computing devices. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0049] The computing device 500 includes a processor 510 (i.e., data processing hardware) that may be used to implement the data processing hardware 12 and / or 62, a memory 520 (i.e., memory hardware) that may be used to implement the memory hardware 14 and / or 64, a storage device 530 (i.e., memory hardware) that may be used to implement the memory hardware 14 and / or 64, a high-speed interface / controller 540 that connects to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 570 that connects to a low-speed bus 560 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or may exist in other manners as needed. The processor 510 may process instructions for execution on the computing device 500, including instructions stored in the memory 520 or the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to the high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as appropriate, and multiple computing devices 500 may be connected, each performing some of the required operations (e.g., as a bank of servers, a group of blade servers, or a multi-processor system).

[0050] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0051] The storage device 530 can provide mass storage for the computing device 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0052] The high-speed controller 540 manages more bandwidth-intensive operations of the computing device 500, and the low-speed controller 570 manages less bandwidth-intensive operations. Such an allocation of roles is merely an example. In some implementations, the high-speed controller 540 is coupled to a memory 520, a display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 570 is coupled to a storage device 530 and a low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and can connect, via a network adapter or the like, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0053] The computing device 500 can be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 500a, or multiple times within a cluster of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0054] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0055] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0056] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0057] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as a field programmable gate array (FPGA), a graphics processing unit (GPU), a tensor processing unit (TPU), or an application specific integrated circuit (ASIC). Processors suitable for executing computer programs include, for example, both general purpose and special purpose processors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0058] Unless expressly stated to the contrary, the phrase "at least one of A, B, or C" is intended to indicate any combination or subset of A, B, and C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A and at least one B, (5) at least one A and at least one C, (6) at least one B and at least one C, and (7) at least one A, at least one B, and at least one C. Additionally, unless expressly stated to the contrary, the phrase "at least one of A, B, and C" is intended to indicate any combination or subset of A, B, and C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A and at least one B, (5) at least one A and at least one C, (6) at least one B and at least one C, and (7) at least one A, at least one B, and at least one C. Additionally, unless expressly stated to the contrary, "A or B" is intended to indicate any combination of A and B, such as (1) A alone, (2) B alone, and (3) A and B.

[0059] To interact with a user, aspects of the invention can be implemented in a computer having a display device, such as a CRT or LCD monitor, or a touch screen, for displaying information to the user, and optionally a keyboard and a pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, verbal, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0060] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.< / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. A combined segmentation and automatic speech recognition (ASR) model (200), An encoder (220), receiving as input a sequence of acoustic frames (110) characterizing one or more utterances (106); an encoder (220) configured to generate, at each of a plurality of output steps, a high-level feature representation (224) of a corresponding acoustic frame (110) in the sequence of acoustic frames (110); and A decoder (260), receiving as input the high-level feature representation (224) generated by the encoder (220) at each of the plurality of output steps; In each of the plurality of output steps, a probability distribution (242) over the possible speech recognition hypotheses; and an indication (212) of whether the corresponding output step corresponds to the end of a speech segment; a decoder (260) configured to generate The joint segmentation and ASR model (200) is trained on a set of training samples, each training sample of the set of training samples having: audio data characterizing the spoken speech; and a corresponding transcription of the uttered utterance, the corresponding transcription having speech segment end ground truth tokens that are automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training samples; The combined segmentation and automatic speech recognition (ASR) model (200) comprises:

2. The decoder (260): A prediction network (300), wherein at each of the plurality of output steps: receiving as input the sequence of non-blank symbols (252) output by the final softmax layer (250); Generate the hidden representation (350) A prediction network (300) configured to A first combining network (210), receiving as input the hidden representation (350) generated by the predictive network (300) at each of the plurality of output steps and the high-level feature representation (224) generated by the encoder (220) at each of the plurality of output steps; At each of the plurality of output steps, generating the indication (212) of whether the corresponding output step corresponds to an end of the speech segment. A first combining network (210) configured to A second combining network (240), receiving as input the hidden representation (350) generated by the predictive network (300) at each of the plurality of output steps and the high-level feature representation (224) generated by the encoder (220) at each of the plurality of output steps; At each of the plurality of output steps, generating the probability distribution (242) over the possible speech recognition hypotheses. A second combining network (240) configured to The combined segmentation and ASR model (200) of claim 1, comprising:

3. In each of the plurality of output steps, the sequence of previous non-blank symbols (252) received as input by the prediction network (300) comprises the sequence of N previous non-blank symbols (252) output by the final softmax layer (250); The prediction network (300) converts the hidden representation (350) into generating a respective embedding (306) for each non-blank symbol (252) of the sequence of N previous non-blank symbols (252); and generating an average embedding (318) by averaging the respective embeddings (306), the average embedding (318) including the hidden representation (350); The combined segmentation and ASR model (200) of claim 2, configured to generate the combined segmentation and ASR model (200) by:

4. The combined segmentation and ASR model of claim 2 , wherein the predictive network includes a V2 embedded lookup table.

5. A training process (130) trains the combined segmentation and ASR model (200) on the set of training samples: training the second combination network (240) to learn how to predict the corresponding transcription of the uttered speech (106) characterized by the audio data of each training sample during a first stage; and After training the second combination network (240), during a second stage: initializing the first combination network (210) with the same parameters as the trained second combination network (240); and using the audio segment termination ground truth tokens inserted into the corresponding transcription of the utterance characterized by the audio data of each training sample. The combined segmentation and ASR model (200) of any one of claims 1 to 4 is trained by:

6. The combined segmentation and ASR model (200) of any of claims 1 to 5, wherein the encoder (220) comprises a causal encoder (220) including a stack of conformer or transformer layers.

7. The combined segmentation and ASR model (200) of any of claims 1 to 6, wherein the audio segment end ground truth tokens are automatically inserted into the corresponding transcriptions without any human annotation.

8. the set of heuristic-based rules and exceptions applied to each training sample in the set of training samples comprises: inserting the ground truth audio segment end token at the end of the corresponding transcription; and adding the ground truth speech segment end token to the corresponding transcription at a location aligned with a non-speech segment of the audio data having a duration that meets a threshold duration; the non-voice segment of the audio data follows a word of the utterance (106) that is identified as a lengthened word; or The non-voiced segment of the audio data follows a word of the uttered utterance (106) that is identified as a filler word. The combined segmentation and ASR model (200) of any of claims 1 to 7, further comprising inserting if not.

9. 9. The combined segmentation and ASR model of claim 8, wherein the word of the spoken utterance is identified as the lengthened word when the phoneme duration of the word meets a standard deviation threshold.

10. 9. The combined segmentation and ASR model of claim 8, wherein after training the combined segmentation and ASR model, the decoder is configured to emit the indication that the corresponding output step corresponds to the end of the speech segment sooner than identifying a number of consecutive non-speech acoustic frames in a sequence of acoustic frames that satisfy the threshold duration.

11. The combined segmentation and ASR model (200) of any of claims 1 to 10, wherein the combined segmentation and ASR model (200) is trained to maximize the probability of emitting the audio segment end ground truth label.

12. A computer-implemented method (400) that, when executed on data processing hardware (510), causes the data processing hardware (510) to perform operations, the operations comprising: receiving as input a sequence of acoustic frames (110) characterizing one or more utterances (106); In each of the multiple output steps, generating, by an encoder (220) of a joint segmentation and automatic speech recognition (ASR) model (200), a high-level feature representation (224) of a corresponding acoustic frame of the sequence of acoustic frames (110); and A decoder (260) of the combined segmentation and ASR model (200) a probability distribution (242) over the possible speech recognition hypotheses; and Indication (212) of whether the corresponding output step corresponds to the end of the speech segment. and Including, The joint segmentation and ASR model (200) is trained on a set of training samples, each training sample of the set of training samples having: audio data characterizing the spoken speech; and a corresponding transcription of the uttered utterance, the corresponding transcription having speech segment end ground truth tokens that are automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training samples; A computer-implemented method (400) comprising:

13. The operation, using a prediction network (300) of the decoder (260) to generate a hidden representation (350) based on the sequence of non-blank symbols (252) output by a final softmax layer (250) in each of the plurality of output steps; generating the indication (212) of whether the corresponding output step corresponds to the end of the speech segment includes using a first combination network (210) of the decoder (260) to generate the indication (212) of whether the corresponding output step corresponds to the end of the speech segment based on the hidden representation (350) generated by the prediction network (300) at each of the plurality of output steps and the high-level feature representation (224) generated by the encoder (220) at each of the plurality of output steps; 13. The computer-implemented method of claim 12, wherein generating a probability distribution (242) over the possible speech recognition hypotheses comprises using a second combination network (240) of the decoder (260) to generate a probability distribution (242) over the possible speech recognition hypotheses based on the hidden representation (350) generated by the prediction network (300) at each of the multiple output steps and the high-level feature representation (224) generated by the encoder (220) at each of the multiple output steps.

14. the sequence of previous non-blank symbols (252) received as input by the prediction network (300) comprises the sequence of N previous non-blank symbols (252) output by the final softmax layer (250); Generating the hidden representation (350) using the prediction network (300) includes: generating a respective embedding (306) for each non-blank symbol (252) of the sequence of N previous non-blank symbols (252); and 14. The computer-implemented method (400) of claim 13, comprising generating an average embedding (318) by averaging the respective embeddings (306), the average embedding (318) including the hidden representation (350).

15. 14. The computer-implemented method (400) of claim 13, wherein the predictive network (300) includes a V2 embedded lookup table.

16. A training process (130) trains the combined segmentation and ASR model (200) on the set of training samples: training the second combination network (240) to learn how to predict the corresponding transcription of the uttered speech (106) characterized by the audio data of each training sample during a first stage; and After training the second combination network (240), during a second stage: initializing the first combination network (210) with the same parameters as the trained second combination network (240); and 16. The computer-implemented method (400) of any of claims 12 to 15, training by using the audio segment end ground truth tokens inserted into the corresponding transcription of the uttered speech characterized by the audio data of each training sample.

17. The computer-implemented method (400) of any of claims 12 to 16, wherein the encoder (220) comprises a causal encoder (220) including a stack of conformer or transformer layers.

18. The computer-implemented method (400) of any of claims 12 to 17, wherein the audio segment end ground truth tokens are automatically inserted into the corresponding transcription without any human annotation.

19. the set of heuristic-based rules and exceptions applied to each training sample in the set of training samples comprises: inserting the ground truth audio segment end token at the end of the corresponding transcription; and adding the ground truth speech segment end token to the corresponding transcription at a location aligned with a non-speech segment of the audio data having a duration that meets a threshold duration; the non-voice segment of the audio data follows a word of the utterance (106) that is identified as a lengthened word; or 19. The computer-implemented method (400) of any of claims 12 to 18, comprising inserting the non-voice segment of the audio data unless it follows a word of the uttered speech (106) that is identified as a filler word.

20. 20. The computer-implemented method (400) of claim 19, wherein the word of the spoken utterance (106) is identified as the lengthened word when the phoneme duration of the word meets a standard deviation threshold.

21. 20. The computer-implemented method (400) of claim 19, wherein after training the joint segmentation and ASR model (200), the operations further include emitting, by the decoder (260), the indication (212) that the corresponding output step corresponds to the end of the speech segment sooner than identifying a number of consecutive non-speech acoustic frames (110) in a sequence of acoustic frames (110) that satisfy the threshold duration.

22. The computer-implemented method (400) of any of claims 12 to 21, wherein the joint segmentation and ASR model (200) is trained to maximize the probability of emitting the audio segment end ground truth label.

Images