Joint segmentation and automatic speech recognition

Abstract

The combined segmentation and ASR model (200) includes an encoder (220) and a decoder (260). The encoder is configured to receive acoustic frames (110) characterizing one or more utterances (106) and generate, at each output step, a high-level feature representation (224) of the corresponding acoustic frame. The decoder is configured to receive the high-level feature representation (224) and generate, at each output step, a probability distribution (242) over possible speech recognition hypotheses and an indication (212) of whether the corresponding output step corresponds to the end of a speech segment. The combined segmentation and ASR model, trained on a set of training samples, each includes audio data characterizing an uttered speech and a corresponding transcription of the uttered speech, where the corresponding transcription automatically inserts a speech segment end ground truth token into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training samples.

Description

[Technical Field]

[0001] TECHNICAL FIELD This disclosure relates to speech segmentation and automatic speech recognition. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems are focused on delivering not only good quality (e.g., low word error rates), but also low latency (e.g., a short delay between when a user speaks and when a transcription appears). For example, when using a device that implements an ASR system today, one often expects the ASR system to decode speech in a streaming format that responds in real time or even faster than real time. Summary of the Invention

[0003] One aspect of the present disclosure provides a combined segmentation and automatic speech recognition (ASR) model including an encoder and a decoder. The encoder is configured to receive as input a sequence of acoustic frames characterizing one or more utterances and generate, at each of a plurality of output steps, a high-order feature representation of a corresponding acoustic frame of the sequence of acoustic frames. The decoder is configured to receive as input the high-order feature representation generated by the encoder at each of the plurality of output steps and generate, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses and an indication of whether the corresponding output step corresponds to the end of a speech segment. The combined segmentation and ASR model is trained with a set of training samples, where each training sample in the set of training samples includes audio data characterizing an uttered speech and a corresponding transcription of the uttered speech, where a speech segment end ground truth token is automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training sample.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the decoder includes a prediction network, a first combination network, and a second combination network. The prediction network is configured to receive as input a sequence of non-blank symbols output by the final softmax layer at each of a plurality of output steps and generate a hidden representation. The first combination network is configured to receive as input the hidden representation generated by the prediction network at each of the plurality of output steps and the high-order feature representation generated by the encoder at each of the plurality of output steps and generate, at each of the plurality of output steps, an indication of whether the corresponding output step corresponds to the end of a speech segment. The second combination network is configured to receive as input the hidden representation generated by the prediction network at each of the plurality of output steps and the high-order feature representation generated by the encoder at each of the plurality of output steps and generate, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses.

[0005] In some examples, in each of the multiple output steps, the sequence of previous non-blank symbols received as input by the prediction network includes a sequence of N previous non-blank symbols output by a final softmax layer. Here, the prediction network is configured to generate a hidden representation by generating a respective embedding for each non-blank symbol in the sequence of N previous non-blank symbols and generating an average embedding by averaging the respective embeddings, where the average embedding includes the hidden representation. In some examples, the prediction network includes a V2 embedding lookup table. In some implementations, the training process trains a combined segmentation and ASR model on a set of training samples by, during a first stage, training a second combination network to learn how to predict corresponding transcriptions of utterances characterized by the audio data of each training sample, and after training the second combination network, during a second stage, initializing the first combination network with the same parameters as the trained second combination network and using audio segment termination ground truth tokens inserted into the corresponding transcriptions of utterances characterized by the audio data of each training sample.

[0006] In some implementations, the encoder comprises a causal encoder including a stack of conformer or transformer layers. In some examples, the audio segment end ground truth tokens are automatically inserted into the corresponding transcription without any human annotation.

[0007] In some examples, the set of heuristic-based rules and exceptions applied to each training sample of the set of training samples includes inserting a ground truth speech segment end token at the end of the corresponding transcription, and inserting the ground truth speech segment end token into the corresponding transcription at a position aligned with a non-speech segment of the audio data having a duration that meets a threshold duration unless the non-speech segment of the audio data follows a word of the utterance that is identified as a lengthened word or the non-speech segment of the audio data follows a word of the utterance that is identified as a filler word. In some implementations, a word of the utterance is identified as a lengthened word when the phoneme duration of the word meets a standard deviation threshold. In some examples, after training the combined segmentation and ASR model, the decoder is configured to emit an indication that the corresponding output step corresponds to the end of the speech segment sooner than identifying the number of consecutive non-speech acoustic frames in the sequence of acoustic frames that meet the threshold duration. In some implementations, the combined segmentation and ASR model is trained to maximize the probability of emitting a speech segment end ground truth label.

[0008] Another aspect of the present disclosure provides a computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations, the operations including receiving as input a sequence of acoustic frames characterizing one or more utterances, and generating, in each of a plurality of output steps, by an encoder of the combined segmentation and automatic speech recognition (ASR) model a high-order feature representation of a corresponding acoustic frame of the sequence of acoustic frames, and by a decoder of the combined segmentation and ASR model a probability distribution over possible speech recognition hypotheses and an indication of whether the corresponding output step corresponds to the end of a speech segment. The combined segmentation and ASR model is trained with a set of training samples, where each training sample of the set of training samples includes audio data characterizing the uttered speech and a corresponding transcription of the uttered speech, where a speech segment end ground truth token is automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training sample.

[0009] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the operations further include using a predictive network of the decoder to generate, at each of the plurality of output steps, a hidden representation based on the sequence of non-blank symbols output by the final softmax layer; generating an indication of whether the corresponding output step corresponds to the end of the speech segment includes using a first combined network of the decoder to generate, at each of the plurality of output steps, an indication of whether the corresponding output step corresponds to the end of the speech segment based on the hidden representation generated by the predictive network at each of the plurality of output steps and the high-order feature representation generated by the encoder at each of the plurality of output steps; and generating a probability distribution over the possible speech recognition hypotheses includes using a second combined network of the decoder to generate, at each of the plurality of output steps, a probability distribution over the possible speech recognition hypotheses based on the hidden representation generated by the predictive network and the high-order feature representation generated by the encoder at each of the plurality of output steps.

[0010] In some examples, the sequence of previous non-blank symbols received as input by the prediction network includes a sequence of N previous non-blank symbols output by a final softmax layer, and generating the hidden representation using the prediction network includes generating the hidden representation by generating a respective embedding for each non-blank symbol in the sequence of N previous non-blank symbols and generating an average embedding by averaging the respective embeddings, where the average embedding includes the hidden representation. In some implementations, the prediction network includes a V2 embedding lookup table. In some examples, the training process trains a combined segmentation and ASR model on a set of training samples by, during a first stage, training a second combination network to learn how to predict corresponding transcriptions of utterances characterized by the audio data of each training sample, and after training the second combination network, during a second stage, initializing the first combination network with the same parameters as the trained second combination network and using audio segment termination ground truth tokens inserted into the corresponding transcriptions of utterances characterized by the audio data of each training sample.

[0011] In some implementations, the encoder includes a causal encoder including a stack of conformer or transformer layers. In some examples, the speech segment end ground truth token is automatically inserted into the corresponding transcription without any human annotation. In some implementations, a set of heuristic-based rules and exceptions applied to each training sample of the set of training samples includes inserting a ground truth speech segment end token at the end of the corresponding transcription, and inserting the ground truth speech segment end token into the corresponding transcription at a position aligned with a non-speech segment of the audio data having a duration that meets a threshold duration unless the non-speech segment of the audio data follows a word of the utterance identified as a lengthened word or the non-speech segment of the audio data follows a word of the utterance identified as a filler word. In some examples, a word of the utterance is identified as a lengthened word when the phoneme duration of the word meets a standard deviation threshold. In some embodiments, after training the combined segmentation and ASR model, the operations also include emitting, by the decoder, an indication that the corresponding output step corresponds to the end of the speech segment sooner than identifying the number of consecutive non-speech audio frames in the sequence of audio frames that meet the threshold duration.

[0012] In some examples, the joint segmentation and ASR model is trained to maximize the probability of emitting an audio segment end ground truth label.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0014] [Figure 1]FIG. 1 is a schematic diagram of an exemplary audio environment using an automatic speech recognition (ASR) system including a speech segment termination combining network for transcribing spoken speech. [Figure 2] FIG. 1 is a schematic diagram of an exemplary ASR system incorporating a speech segment termination combining network. [Figure 3] FIG. 3 is a schematic diagram of an exemplary concatenation and reduction prediction network of the ASR system of FIG. 2. [Figure 4] 1 is a flowchart of an exemplary configuration of operations of a computer-implemented method for executing an ASR system including a speech segment termination model. [Figure 5] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] Improving the performance of end-to-end (E2E) ASR models for long utterances ranging in length from minutes to hours is a persistent challenge in speech recognition. Streaming E2E ASR models have achieved low word error rates (WER) for short- to medium-length utterances up to a few minutes in length. However, traditional E2E ASR models can have high WERs and suffer from deletion errors for long-form utterances ranging from tens of minutes to hours in length. Such utterances are found, for example, in tasks such as meeting and lecture transcription and video captioning.

[0017] Traditionally, long-form speech is first segmented using a separate voice activity detector (VAD). Whenever the VAD detects a long silence, it splits the audio at that location into two segments, which are then processed separately, or independently, by the E2E ASR model. At each segment boundary, the beam search finalizes the top hypothesis by discarding all other hypotheses. This introduces more diversity into the beam search by occasionally wiping out old hypotheses to make room for new ones, ultimately improving WER by confirming more potentially correct hypotheses. Maintaining beam diversity is particularly important for E2E ASR models, which are typically decoded with small beams. However, VAD-based segmentators, by design, suffer from high latency because the VAD must wait through long silences before making a segmentation decision. This delays subsequent ASR functions, such as rescoring or prefetching, which must wait for the top hypothesis to finish. Improving latency is important because it can improve the user experience by improving the responsiveness of smart assistants through faster prefetching, or by helping dictation or caption apps reduce the amount of "flickering" caused by switching between top hypotheses. Current VAD-based segmentators also suffer from high segmentation errors because VAD bases its decisions purely on audio rather than decoded text, which may contain semantic cues about when to segment. Improving segmentation accuracy is important because it can improve WER. On the other hand, not segmenting at all leads to an inflated beam search with no hypothesis diversity, which can also induce word errors.

[0018] Embodiments herein are directed toward integrating an ASR system (e.g., including a recurrent neural network-transducer (RNN-T) model) with a speech segment termination model (e.g., an additional combination network) configured and trained to predict whether a corresponding output step of the ASR system corresponds to the end of a speech segment event based on acoustic and semantic features. The disclosed ASR system is jointly optimized for both speech segment termination detection and ASR tasks. However, existing long utterance training data for ASR systems typically does not include ground truth labels corresponding to speech segment terminations, making supervised training of speech segment termination combinations challenging. Embodiments herein are directed toward implementing a speech segment termination annotation routine for automatically inserting speech segment termination ground truth labels into ground truth training transcripts based on modeling hesitation and word timing. An exemplary speech segment termination scheme applies a heuristic-based, weakly supervised approach, in which speech segment termination ground truth labels are automatically inserted into ground truth training transcripts by applying one or more rules.

[0019] FIG. 1 is a schematic diagram of an example system 100 and an audio environment 102. In the audio environment 102, a user 104's way of interacting with a computing device, such as a user device 10, can be via voice input. The user device 10 (also commonly referred to as device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within the audio environment 102. Here, streaming audio data can refer to voiced utterances 106 by the users 104 that function as audible queries, commands to the device 10, or audible communications captured by the device 10. The voice-enabled system of the device 10 can address the queries or commands by responding to the queries and / or causing the commands to be executed / accomplished by one or more downstream applications. The voice-enabled system of the device 10 can also generate transcriptions of the audible communications captured by the device 10.

[0020] The user device 10 is associated with a user 104 and may correspond to any computing device capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, in-vehicle infotainment systems, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12. The memory hardware 14 stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 comprising audio capture devices (e.g., microphones) 16, 16a for capturing and converting uttered speech 106 into electrical signals within the audio environment 102, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the user device 10). In the illustrated example, the user device 10 implements a single audio capture device 16a, but the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, such that one or more capture devices 16a of the array may communicate with the audio system 16 without being physically present at the user device 10.

[0021] In the system 100, the automatic speech recognition (ASR) system 118 includes an RNN-T model 200 that implements a speech segment termination model 205 and resides on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system running in a cloud computing environment) that communicates with the user device 10 via the network 40. The user device 10 and / or the remote computing device 60 also includes an audio subsystem 108 that is configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16a and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 118. In the illustrated example, the user speaks each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., acoustic frames) 110 for input to the ASR system 118. The RNN-T model 200 then receives as input an acoustic frame 110 corresponding to the utterance 106, generates an indication of whether the corresponding output step corresponds to the end of a speech segment that can be used to divide the utterance 106 into speech segments, and generates / predicts as output a transcription 120 corresponding to each speech segment of the utterance 106. In the illustrated example, the RNN-T model 200 can perform streaming speech recognition to generate partial transcriptions (e.g., streaming speech recognition results) 120a, 120b as the user speaks.

[0022] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120, 120a-n of the utterance 106 to the user 104 of the user device 10 and to present one or more responses 121, 121a-n to the queries and / or commands of the utterance 106. As described in more detail below, the user interface generator 107 may display the speech recognition results 120, 120a-n and the responses 121, 121a-n over time or at different times. As shown, the user interface generator 107 may present the speech recognition results 120, 120a-n (e.g., transcriptions) and responses 121, 121a-n to represent an interaction / conversation between the user 104 and an interactive program or application (e.g., digital assistant application 50).

[0023] In some configurations, the transcription 120 output from the ASR system 118 for each identified speech segment is processed, for example, by a natural language processing / understanding (NLP / NLU) module running on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. The digital assistant application 50 may provide an appropriate response 121 after executing the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., running on any combination of the user device 10 or the remote computing device 60) may convert the transcription 120 and / or the response 121 into audible synthesized speech output by the user device 10 and / or another device.

[0024] In the illustrated example, a user 104 interacts with a program or application (e.g., a digital assistant application 50) on a user device 10 that uses an ASR system 118. For example, FIG. 1 shows a user 104 communicating with a digital assistant application 50 and the digital assistant application 50 displayed in an interactive digital assistant interface 18 on a screen 19 of the user device 10 to illustrate a natural conversational interaction between the user 104 and the digital assistant application 50. In this example, the user 104 utters a first speech segment 106a of an utterance 106 ("Dictate my next message") and then utters a second speech segment 106b of the utterance 106 ("Where can I order Thai food?").

[0025] When the ASR system 118 receives acoustic frames 110 corresponding to utterances 106 as uttered by the user 104, the speech segment termination model 205 generates, at each output step, an indication of whether the output step corresponds to the end of a speech segment. The speech segment termination indications generated by the speech segment termination model 205 can be used to delineate the speech segments 106 of the utterance 106, and the RNN-T model 200 can perform speech recognition on each speech segment 106 individually. In the illustrated example, the speech segment termination model 205 detects or determines a speech segment termination between a first speech segment 106 and a second speech segment 106 and outputs a speech segment termination indication indicating the end of the first speech segment 106a.

[0026] For each speech segment 106 of the utterance 106 indicated or identified by the speech segment termination model 205, the RNN-T model 200 performs speech recognition on the acoustic frames 110 corresponding to the speech segment 106 to generate a speech recognition result 120 corresponding to the speech segment 106. Continuing with this example, based on the speech segment termination model 205 detecting an ending segment of speech between the first speech segment 106 and the second speech segment 106, at time 1, the user interface generator 107 presents, via the digital assistant interface 18, a display of the first speech recognition result 120a for the first speech segment 106a to the user 104 of the user device 10 in a streaming format, whereby words, word pieces, and / or individual characters are displayed on the screen 19 of the user device 10 as soon as they are uttered.

[0027] Continuing with this example, based on the speech segment termination model 205 detecting the termination segment of speech for the second speech segment 106, at time 2, the user interface generator 107 presents a display of the second speech recognition result 120b for the second speech segment 106b to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming format, whereby words, word pieces, and / or individual characters are displayed on the screen 19 of the user device 10 as soon as they are uttered. Then, during time 3, the user interface generator 107 presents via the digital assistant interface 18 a response 121b (i.e., "Send a message?") generated by the assistant application 50 based on the speech recognition results 120a, 120b generated for the speech segments 106a, 106b.

[0028] 2 is a schematic diagram of an exemplary RNN-T model 200 that integrates a speech segment termination model 205 as a first combination network 210 for detecting boundaries between uttered speech segments. As shown, the RNN-T model 200 includes an encoder network 220, a prediction network 300, a second combination network 240 (i.e., wordpiece combination network 240), and a final softmax output layer 250. The encoder network 220 (e.g., an audio encoder), which is broadly similar to an acoustic model (AM) in a conventional ASR system, generates a sequence of feature vectors.

number

number

number

[0029] In the illustrated example, the prediction network 300 includes an LSTM-based prediction network, which, like a language model (LM), calculates the sequence of non-empty symbols previously output by the softmax layer 250.

number

number

number

[0030] Figure 3 shows the input for the non-whitespace symbol y u-n ,...,y u-1 1 illustrates a prediction network 300 for an RNN-T model 200 that receives a sequence of non-blank symbols 252a-n, which is limited to the N previous non-blank symbols 252a-n output by the final softmax layer 250. In some examples, N is equal to 2. In other examples, N is equal to 5, although this disclosure is non-limiting and N may be equal to any integer. The sequence of non-blank symbols 252a-n may represent a partial speech recognition result 120a, 120b (FIG. 1). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A-H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads may be used by the multi-head attention mechanism 302. Notably, the multi-head attention mechanism 302 significantly improves performance with a minimal increase in model size. As explained in more detail below, each head 302A-H contains its own row of position vectors 308, and rather than increasing the model size by concatenating the outputs 318A-H from all heads, the outputs 318A-H are instead averaged by a head averaging module 322.

[0031] Referring to the first head 302A of the multi-head attention mechanism 302, head 302A uses a shared embedding matrix 304 to generate a sequence of non-empty symbols y received as input at a corresponding output step from multiple output steps. ui-n , ..., y ui-1 For each non-blank symbol 301 in

number

number

[0032] a sequence of non-blank symbols 252a-n, y un,..., y The corresponding embedding generated by the shared embedding matrix 304 for each non-blank symbol 301 in u-1 is the same for all of the heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines a position vector PV Aa~An The second head 302B defines a position vector PV Ba~Bn 308 of Ba~Bn , ..., the Hth head 302H defines a position vector PV Ha~Hn 308 of Ha~Hn Define another different line of

[0033] For each non-blank symbol in the received sequence of non-blank symbols 252a-n, the first head 302A, via the weight layer 310, weights the corresponding embedding 306 in proportion to the similarity between the corresponding embedding and its assigned respective position vector PV 308. In some examples, the similarity may include cosine similarity (e.g., cosine distance). In the example shown, the weight layer 310 outputs a sequence of weighted embeddings 312, 312Aa-An, each weighting the corresponding embedding 306 in proportion to its assigned respective position vector PV 308. Stated differently, the weighted embedding 312 output by the weight layer 310 for each embedding 306 may correspond to the dot product between the embedding 306 and its assigned respective position vector PV 308. The weighted embedding 312 may be interpreted as overriding the embedding in proportion to how similar the embedding is to the positioning associated with its assigned respective position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, and therefore the sequence of weighted embeddings 312Aa-An is not concatenated but instead averaged by a weighted average module 316 to produce a weighted average 318A of the weighted embeddings 312Aa-An as output from the first head 302A, expressed as follows:

number

[0034] The operations described above with respect to the first head 302A are similarly performed for each of the other heads 302B-H of the multi-head attention mechanism 302. Due to the different sets of positioning vectors PV 308 defined by each head 302, the weight layer 310 outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn for each of the other heads 302B-H that differs from the sequence of weighted embeddings 312Aa-Aa for the first head 302A. The weighted average module 316 then generates a weighted average 318B-H of each of the corresponding weighted embeddings 312 of the sequences of non-blank symbols as output from the corresponding heads 302B-H.

[0035] In the example shown, the prediction network 300 includes a head averaging module 322 that averages weighted averages 318A-H output from corresponding heads 302A-H. A projection layer 326 with SWISH may receive as input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H and generate as output a projection output 328. A final layer normalization 330 normalizes the projection output 328 to produce a single embedding vector at the corresponding output step from multiple output steps.

number

number

[0036] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302, and only performs the operations described above for the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer 326 and layer normalization 330 to produce a single embedding vector

number

[0037] Referring back to FIG. 2A, the speech segment termination combining network 210 (i.e., the first combining network 210) extracts a single embedding vector (i.e., a sigma-based embedding vector) from the prediction network 300.

number

number

number

number

number

number

[0038] In some implementations, the training data generator 140 (see FIG. 1) uses a heuristic-based, weakly supervised approach to generate ground truth <eos>Automatically insert tokens into paired audio transcription training data (e.g., without any human annotations) to generate ground truth <eos>Tokens are automatically inserted based on one or more rules. Exemplary rules include, but are not limited to, matching ground truth audio segment ends with the end of the corresponding transcription. <eos>Insert tokens and ground truth audio segment terminations <eos>The method includes inserting a token into the corresponding transcription at a position aligned with a non-speech segment (e.g., silence) of the audio data having a duration that meets a threshold duration, unless the non-speech segment of the audio data follows a word of the spoken utterance identified as a lengthened word (e.g., "heyyy") or follows a word of the spoken utterance identified as a filler word (e.g., "um"). Such exceptions reflect situations in which a speaker is likely not finished uttering a speech segment. A lengthened word may be detected, for example, when a phoneme duration meets a standard deviation threshold (e.g., when the phoneme duration is greater than five times the standard deviation of the phoneme duration). Filler words may be detected using a filler word detection model. In some implementations, the training data generator 140 applies the rules and exceptions by capturing the timing of silence, words, and phonemes using a forced alignment model applied to all audio transcription pairs in the training dataset.

[0039] The wordpiece combination network 240 (i.e., the second combination network 240) extracts a single embedding vector from the prediction network 300.

number

number

number

number

number

number

[0040] A speech segment termination combining network 210 detects the termination of a speech segment (e.g., <eos>Given a representation of the token 212, the softmax layer 250 calculates the next output symbol predicted by the RNN-T model 200 at the corresponding output step.

number

number

[0041] In some implementations, the feature vectors input to the encoder network 220 are

number

[0042] The prediction network 300 may include an LTSM-based network with an embedding dimension of 320.

number

number

[0043] The speech segment termination combining network 210, the wordpiece combining network 240, and the prediction network 300 may collectively form the RNN-T decoder 260 of the RNN-T model 200. In some implementations, to further reduce the size of the RNN-T decoder 260, i.e., the prediction network 300 and the wordpiece combining network 240, parameter combining between the prediction network 300 and the wordpiece combining network 240 is applied. Specifically, the vocabulary size |V| and the embedding dimension d e , then the shared embedding matrix 304 in the prediction network is

number

number

[0044] In some implementations, the RNN-T model 200 performs a frame-synchronous beam search using a beam size of 8 and a pruning threshold of 5 (i.e., negative hypotheses whose negative log posterior exceeds that of the top hypothesis by 5 are removed). At each frame, the RNN-T model 200 performs a breadth-first search for possible expansions, ignoring any expansions with a negative log posterior of 5 or more, and limiting the search depth to 10 expansions. In some examples, the RNN-T model 200 enforces the termination of speech segments exceeding 65 seconds in duration.

[0045] 4 is a flowchart of an exemplary operational arrangement of a computer-implemented method 400 for performing joint segmentation and automatic speech recognition. At operation 402, the method 400 generates a sequence of acoustic frames (e.g.,

number

[0046] For each of the plurality of output steps, the method 400 performs operations 404, 406, and 408. In operation 404, the method 400 generates, for each of the plurality of output steps, a high-dimensional feature representation of the corresponding acoustic frame of the sequence of acoustic frames by the encoder network 220 of a joint segmentation and automatic speech recognition (ASR) model (e.g., RNN-T200).

number

[0047] In operation 406, the method 400 generates a probability distribution over the possible speech recognition hypotheses by the decoder 260 of the joint segmentation and ASR model.

number

[0048] 5 is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computing devices. The components shown here, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the scope of the invention as described and / or claimed in this document.

[0049] The computing device 500 includes a processor 510 (i.e., data processing hardware) that may be used to implement the data processing hardware 12 and / or 62, a memory 520 (i.e., memory hardware) that may be used to implement the memory hardware 14 and / or 64, a storage device 530 (i.e., memory hardware) that may be used to implement the memory hardware 14 and / or 64, a high-speed interface / controller 540 that connects to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 570 that connects to a low-speed bus 560 and the storage device 530. The components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or exist in other manners as needed. The processor 510 processes instructions for execution on the computing device 500, including instructions stored in the memory 520 or the storage device 530, and can display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to the high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as needed, and multiple computing devices 500 may be connected, each performing some of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0050] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0051] Storage device 530 can provide mass storage for computing device 500. In some embodiments, storage device 530 is a computer-readable medium. In various different implementations, storage device 530 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0052] The high-speed controller 540 manages more bandwidth-intensive operations of the computing device 500, while the low-speed controller 570 manages less bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, the high-speed controller 540 is coupled to memory 520, a display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 570 is coupled to a storage device 530 and a low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and may connect, via a network adapter or the like, to one or more input / output devices such as a keyboard, pointing device, scanner, or network devices such as a switch or router.

[0053] The computing device 500 can be implemented in many different forms, as shown in the figure. For example, it may be implemented as a standard server 500a, or multiple times within a cluster of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0054] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general-purpose, and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0055] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0056] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0057] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special-purpose logic circuitry, such as an FPGA (field-programmable gate array), a graphics processing unit (GPU), a tensor processing unit (TPU), or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose processors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0058] Unless expressly stated to the contrary, the phrase "at least one of A, B, or C" is intended to indicate any combination or subset of A, B, and C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A and at least one B, (5) at least one A and at least one C, (6) at least one B and at least one C, and (7) at least one A, at least one B, and at least one C. Furthermore, unless expressly stated to the contrary, the phrase "at least one of A, B, and C" is intended to indicate any combination or subset of A, B, and C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A and at least one B, (5) at least one A and at least one C, (6) at least one B and at least one C, and (7) at least one A, at least one B, and at least one C. Furthermore, unless expressly stated to the contrary, "A or B" is intended to indicate any combination of A and B, such as (1) A alone, (2) B alone, and (3) A and B.

[0059] To interact with a user, aspects of the present invention can be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen, for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, verbal, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to devices used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0060] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.< / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. A combined segmentation and automatic speech recognition (ASR) model, comprising:

1. An encoder comprising: receiving as input a sequence of acoustic frames characterizing one or more utterances; an encoder configured to generate, at each of a plurality of output steps, a high-order feature representation of a corresponding acoustic frame in the sequence of acoustic frames; and A decoder comprising: receiving as input the high-dimensional feature representations generated by the encoder at each of the plurality of output steps; In each of the plurality of output steps, a probability distribution over the possible speech recognition hypotheses; and an indication of whether the corresponding output step corresponds to the end of an audio segment; a decoder configured to generate The joint segmentation and ASR model is trained on a set of training samples, each training sample of the set of training samples being: audio data characterizing the spoken speech; and a corresponding transcription of the uttered speech, the corresponding transcription having a ground truth end of speech segment <eos> token automatically inserted into the corresponding transcription by inserting a ground truth end of speech segment <eos> token at the end of the corresponding transcription and inserting a ground truth end of speech segment <eos> token in a position aligned with a non-speech segment of audio data having a duration that meets a threshold duration in the corresponding transcription, except when the non-speech segment of the audio data follows a word in the utterance that is identified as a lengthened word or when the non-speech segment of the audio data follows a word in the utterance that is identified as a filler word; A combined segmentation and ASR model, including:

2. The decoder: A prediction network, wherein at each of the plurality of output steps: It receives as input the sequence of non-blank symbols output by the final softmax layer, Generate a hidden representation a prediction network configured as follows: a first coupling network, receiving as input the hidden representation generated by the predictive network at each of the plurality of output steps and the high-level feature representation generated by the encoder at each of the plurality of output steps; and generating, at each of the plurality of output steps, the indication of whether the corresponding output step corresponds to the end of the speech segment. a first coupling network configured to a second coupling network, receiving as input the hidden representation generated by the predictive network at each of the plurality of output steps and the high-level feature representation generated by the encoder at each of the plurality of output steps; generating the probability distribution over the possible speech recognition hypotheses at each of the plurality of output steps; a second coupling network configured as The combined segmentation and ASR model of claim 1 , comprising:

3. In each of the plurality of output steps, the sequence of previous non-blank symbols received as input by the prediction network includes the sequence of N previous non-blank symbols output by the final softmax layer; The predictive network converts the hidden representation into generating a respective embedding for each non-blank symbol in the sequence of N previous non-blank symbols; and generating an average embedding by averaging the respective embeddings, the average embedding including the hidden representation; The combined segmentation and ASR model of claim 2 , configured to generate the combined segmentation and ASR model by:

4. The combined segmentation and ASR model of claim 2 , wherein the predictive network includes a V2 embedded lookup table.

5. a training process for training the combined segmentation and ASR model on the set of training samples; training the second combination network during the first stage to learn how to predict the corresponding transcription of the uttered speech characterized by the audio data of each training sample; and After training the second combination network, during a second stage: initializing the first combination network with the same parameters as the trained second combination network; and using the audio segment termination ground truth tokens inserted into the corresponding transcription of the uttered utterance characterized by the audio data of each training sample. The combined segmentation and ASR model of claim 2 is trained by:

6. The combined segmentation and ASR model of claim 1 , wherein the encoder comprises a causal encoder comprising a stack of conformer or transformer layers.

7. 2. The combined segmentation and ASR model of claim 1, wherein the audio segment end ground truth tokens are automatically inserted into the corresponding transcription without any human annotation.

8. 8. The combined segmentation and ASR model of claim 7, wherein the word of the spoken utterance is identified as the lengthened word when the phoneme duration of the word meets a standard deviation threshold.

9. 8. The combined segmentation and ASR model of claim 7, wherein after training the combined segmentation and ASR model, the decoder is configured to emit the indication that the corresponding output step corresponds to the end of the speech segment sooner than identifying a number of consecutive non-speech audio frames in a sequence of audio frames that meets the threshold duration.

10. The combined segmentation and ASR model of claim 1 , wherein the combined segmentation and ASR model is trained to maximize the probability of emitting an audio segment end ground truth label.

11. A method implemented by data processing hardware, comprising: receiving as input a sequence of acoustic frames characterizing one or more utterances; In each of the multiple output steps, generating a high-order feature representation of a corresponding acoustic frame of the sequence of acoustic frames by an encoder of a combined segmentation and automatic speech recognition (ASR) model; and a decoder of the combined segmentation and ASR model, a probability distribution over the possible speech recognition hypotheses; and an indication of whether the corresponding output step corresponds to the end of an audio segment; and Including, The joint segmentation and ASR model is trained on a set of training samples, each training sample of the set of training samples being: audio data characterizing the spoken speech; and a corresponding transcription of the uttered speech, the corresponding transcription having a ground truth end of speech segment <eos> token automatically inserted into the corresponding transcription by inserting a ground truth end of speech segment <eos> token at the end of the corresponding transcription and inserting a ground truth end of speech segment <eos> token in a position aligned with a non-speech segment of audio data having a duration that meets a threshold duration in the corresponding transcription, except when the non-speech segment of the audio data follows a word in the utterance that is identified as a lengthened word or when the non-speech segment of the audio data follows a word in the utterance that is identified as a filler word; A method comprising:

12. The method comprises: further comprising using a predictive network of the decoder to generate a hidden representation based on the sequence of non-blank symbols output by a final softmax layer in each of the plurality of output steps; generating the indication of whether the corresponding output step corresponds to the end of the speech segment includes using a first combinational network of the decoder to generate the indication of whether the corresponding output step corresponds to the end of the speech segment based on the hidden representation generated by the prediction network at each of the plurality of output steps and the high-level feature representation generated by the encoder at each of the plurality of output steps; 12. The method of claim 11 , wherein generating the probability distribution over the possible speech recognition hypotheses comprises using a second combinational network of the decoder to generate a probability distribution over the possible speech recognition hypotheses based on the hidden representations produced by the prediction network at each of the plurality of output steps and the high-level feature representations produced by the encoder at each of the plurality of output steps.

13. the sequence of previous non-blank symbols received as input by the prediction network includes the sequence of N previous non-blank symbols output by the final softmax layer; Generating the hidden representation using the predictive network comprises: generating a respective embedding for each non-blank symbol in the sequence of N previous non-blank symbols; and 13. The method of claim 12, comprising generating an average embedding by averaging the respective embeddings, the average embedding including the hidden representation.

14. The method of claim 12 , wherein the predictive network includes a V2 embedded lookup table.

15. a training process for training the combined segmentation and ASR model on the set of training samples; training the second combination network during the first stage to learn how to predict the corresponding transcription of the uttered speech characterized by the audio data of each training sample; and After training the second combination network, during a second stage: initializing the first combination network with the same parameters as the trained second combination network; and 13. The method of claim 12, training by using the audio segment end ground truth tokens inserted into the corresponding transcription of the uttered speech characterized by the audio data of each training sample.

16. The method of claim 11 , wherein the encoder comprises a causal encoder comprising a stack of conformer or transformer layers.

17. The method of claim 11 , wherein the audio segment end ground truth tokens are automatically inserted into the corresponding transcription without any human annotation.

18. The method of claim 11 , wherein the word of the spoken utterance is identified as the lengthened word when the phoneme duration of the word meets a standard deviation threshold.

19. 12. The method of claim 11, wherein after training the joint segmentation and ASR model, the method further comprises emitting, by the decoder, the indication that the corresponding output step corresponds to the end of the speech segment sooner than identifying a number of consecutive non-speech sound frames in a sequence of sound frames that meets the threshold duration.

20. The method of claim 11 , wherein the joint segmentation and ASR model is trained to maximize the probability of emitting an audio segment end ground truth label.

Images