Joint segmentation and automatic speech recognition

Abstract

A joint segmentation and ASR model (200) includes an encoder (220) and a decoder (260). The encoder is configured to: receive an acoustic frame (110) characterizing one or more utterances (106); and generating a higher order feature representation (224) for the corresponding acoustic frame in each output step. The decoder is configured to: receive the higher order feature representation (224); and generating in each output step: a probability distribution of possible speech recognition hypotheses (242); and an indication (212) as to whether the corresponding output step corresponds to the end of the speech segment. The joint segmentation and ASR model is trained by using a group of training samples, and each training sample comprises audio data representing a spoken utterance; and a corresponding transcription of the spoken utterance, the corresponding transcription having a speech segment end true value lemma, the speech segment end true value lemma being automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training sample.

Description

Technical Field

[0001] The present disclosure relates to speech segmentation and automatic speech recognition. Background Art

[0002] Modern automatic speech recognition (ASR) systems focus not only on providing high quality (e.g., low word error rate), but also on providing low latency (e.g., a short delay between the user speaking and the transcription appearing). For example, when using a device implementing an ASR system today, it is often expected that the ASR system decodes the utterance in a streaming manner corresponding to real-time or even faster than real-time. Summary of the invention

[0003] One aspect of the present disclosure provides a joint segmentation and automatic speech recognition (ASR) model, which includes an encoder and a decoder. The encoder is configured to: receive as input a sequence of acoustic frames representing one or more utterances; and in each of a plurality of output steps, generate a high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The decoder is configured to: receive as input a high-order feature representation generated by the encoder in each of the plurality of output steps; and in each of the plurality of output steps, generate: a probability distribution of possible speech recognition hypotheses; and an indication of whether the corresponding output step corresponds to the end of a speech segment. The joint segmentation and ASR model is trained with a set of training samples, each of which includes: audio data representing a spoken utterance; and a corresponding transcription of the spoken utterance, the corresponding transcription having a speech segment end real value token, which is automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training sample.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the decoder includes a prediction network, a first joint network, and a second joint network. The prediction network is configured to: receive as input a sequence of non-blank symbols output by the last softmax layer in each of the multiple output steps; and generate a hidden representation. The first joint network is configured to: receive as input a hidden representation generated by the prediction network in each of the multiple output steps and a high-order feature representation generated by the encoder in each of the multiple output steps; and generate in each of the multiple output steps an indication as to whether the corresponding output step corresponds to the end of a speech segment. The second joint network is configured to: receive as input a hidden representation generated by the prediction network in each of the multiple output steps and a high-order feature representation generated by the encoder in each of the multiple output steps; and in each of the multiple output steps, generate a probability distribution of possible speech recognition hypotheses.

[0005] In some examples, in each of the multiple output steps, the sequence of previous non-blank symbols received as input at the prediction network includes a sequence of N previous non-blank symbols output by the last softmax layer. Here, the prediction network is configured to generate a hidden representation by the following operations: for each non-blank symbol in the sequence of N previous non-blank symbols, a corresponding embedding is generated; and an average embedding is generated by averaging the corresponding embeddings, the average embedding including the hidden representation. In some examples, the prediction network includes a V2 embedding lookup table. In some implementations, the training process trains the joint segmentation and ASR model with the set of training samples by the following operations: during the first stage, a second joint network is trained to learn how to predict the corresponding transcription of the spoken utterance represented by the audio data of each training sample; and after training the second joint network, during the second stage, the first joint network is initialized using the same parameters as the trained second joint network, and the speech segment end real value tokens inserted into the corresponding transcription of the spoken utterance represented by the audio data of each training sample are used.

[0006] In some implementations, the encoder comprises a causal encoder comprising a stack of a conformer layer or a transformer layer.In some examples, the end-of-segment real-value token is automatically inserted into the corresponding transcription without any manual annotation.

[0007] In some examples, the set of heuristic-based rules and exceptions applied to each training sample in the set of training samples includes: inserting a true value speech segment end token at the end of the corresponding transcription; and inserting a true value speech segment end token into the corresponding transcription at a position aligned with a non-speech segment of the audio data, the non-speech segment having a duration that satisfies a threshold duration, unless the non-speech segment of the audio data follows a word identified as a lengthened word in the spoken utterance, or the non-speech segment of the audio data follows a word identified as a filler word in the spoken utterance. In some implementations, a word in the spoken utterance is identified as a lengthened word when the phoneme duration of the word in the spoken utterance satisfies a standard deviation threshold. In some examples, after training the joint segmentation and ASR model, the decoder is configured to issue an indication that the corresponding output step corresponds to the end of the speech segment earlier than a plurality of consecutive non-speech acoustic frames that meet the threshold duration in the sequence of identified acoustic frames. In some implementations, the joint segmentation and ASR model is trained to maximize the probability of emitting a true value label of the end of the speech segment.

[0008] Another aspect of the present disclosure provides a computer-implemented method executed on data processing hardware, thereby causing the data processing hardware to perform operations, the operations comprising: receiving a sequence of acoustic frames representing one or more utterances; and in each of the plurality of output steps: generating, by an encoder of a joint segmentation and automatic speech recognition (ASR) model, a high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; and generating, by a decoder of the joint segmentation and ASR model: a probability distribution over possible speech recognition hypotheses; and an indication as to whether the corresponding output step corresponds to an end of speech segment. The joint segmentation and ASR model is trained with a set of training samples, each training sample in the set of training samples comprising: audio data representing a spoken utterance; and a corresponding transcription of the spoken utterance, the corresponding transcription having an end-of-segment truth-value token automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training sample.

[0009] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation also includes generating a hidden representation based on a sequence of non-blank symbols output by the last softmax layer using a prediction network of a decoder in each of the multiple output steps; generating an indication as to whether the corresponding output step corresponds to the end of a speech segment includes: using a first joint network of the decoder, based on a hidden representation generated by the prediction network in each of the multiple output steps and a high-order feature representation generated by the encoder in each of the multiple output steps to generate an indication as to whether the corresponding output step corresponds to the end of a speech segment; and generating a probability distribution of possible speech recognition hypotheses includes: using a second joint network of the decoder, based on a hidden representation generated by the prediction network in each of the multiple output steps and a high-order feature representation generated by the encoder in each of the multiple output steps to generate a probability distribution of possible speech recognition hypotheses.

[0010] In some examples, the sequence of previous non-blank symbols received as input at the prediction network includes a sequence of N previous non-blank symbols output by the last softmax layer; and generating a hidden representation using the prediction network includes generating the hidden representation by: for each non-blank symbol in the sequence of N previous non-blank symbols, generating a corresponding embedding; and generating an average embedding by averaging the corresponding embeddings, the average embedding comprising the hidden representation. In some implementations, the prediction network includes a V2 embedding lookup table. In some examples, the training process trains the joint segmentation and ASR model with the set of training samples by: during a first stage, training a second joint network to learn how to predict a corresponding transcription of a spoken utterance represented by the audio data of each training sample; and after training the second joint network, during a second stage: initializing the first joint network using the same parameters as the trained second joint network; and using speech segment end real value tokens inserted into the corresponding transcription of the spoken utterance represented by the audio data of each training sample.

[0011] In some implementations, the encoder includes a causal encoder that includes a stack of conformer layers or transformer layers. In some examples, the speech segment end real-value tokens are automatically inserted into the corresponding transcription without any manual annotation. In some implementations, the set of heuristic-based rules and exceptions applied to each training sample in the set of training samples includes: inserting the real-value speech segment end token at the end of the corresponding transcription; and inserting the real-value speech segment end token into the corresponding transcription at a position aligned with a non-speech segment of the audio data, the non-speech segment having a duration that meets a threshold duration, unless the non-speech segment of the audio data follows a word identified as a lengthened word in the spoken utterance, or the non-speech segment of the audio data follows a word identified as a filler word in the spoken utterance. In some examples, a word in the spoken utterance is identified as a lengthened word when the phoneme duration of the word in the spoken utterance meets a standard deviation threshold. In some implementations, after training the joint segmentation and ASR model, the operation further includes: earlier than a plurality of consecutive non-speech acoustic frames in the sequence of identified acoustic frames that meet a threshold duration, the decoder issues an indication that the corresponding output step corresponds to the end of the speech segment.

[0012] In some examples, the joint segmentation and ASR model is trained to maximize the probability of uttering a true value label of the end of speech segment.

[0013] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of an example speech environment using an automatic speech recognition (ASR) system including a speech segment end association network for transcribing spoken utterances.

[0015] Figure 2 is a schematic diagram of an example ASR system integrating a speech segment end joint network.

[0016] Figure 3 yes Figure 2 Example bundling of an ASR system and a schematic diagram of the simplified prediction network.

[0017] Figure 4 is a flow diagram of an example arrangement of operations for a computer-implemented method of performing an ASR system including an end-of-segment model.

[0018] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0019] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0020] Improving the performance of end-to-end (E2E) ASR models for long utterances of several minutes to several hours in length is an ongoing challenge in speech recognition. Streaming E2E ASR models have achieved low word error rates (WER) for short to medium length utterances of up to several minutes in length. However, for long utterances of tens of minutes to tens of hours in length, conventional E2E ASR models may have high WER and make deletion errors. Such utterances are seen in tasks such as transcribing meetings and lectures and adding subtitles to videos.

[0021] Conventionally, long utterances are first segmented using a separate voice activity detector (VAD). Whenever the VAD detects a long period of silence, it splits the audio into two segments at that location, and then processes the two segments separately or independently by the E2E ASR model. At each segment boundary, the beam search finally determines the best hypothesis by discarding all other hypotheses. This introduces more diversity into the beam search by occasionally clearing out old hypotheses and making room for new hypotheses, and ultimately improves the WER by looking at more possible correct hypotheses. Maintaining beam diversity is particularly important for E2E ASR models, which are usually decoded using small beams. However, VAD-based segmenters have high latency because, by design, the VAD must wait for long periods of silence before deciding on a segment. This delays subsequent ASR functions that must wait until the best hypothesis is finally determined, such as rescoring or prefetching. Improving latency is important because it can improve the user experience by making the intelligent assistant respond faster via faster prefetching or by helping dictation or subtitle applications reduce the amount of "flickering" that occurs due to switching between the best hypotheses. Current VAD-based segmenters also suffer from high segmentation errors, since VAD makes decisions purely based on the audio and not the decoded text, which may contain semantic clues about when to segment. Improving segmentation correctness is important because it improves WER. On the other hand, not segmenting at all will lead to beam search inflation and non-diversity assumptions, which may also lead to word errors.

[0022] The implementations herein are intended to integrate an ASR system (e.g., including a Recurrent Neural Network-Transformer (RNN-T) model) with a speech segment end model (e.g., an additional joint network), which is configured and trained to predict whether a corresponding output step of the ASR system corresponds to a speech segment end event based on acoustic features and semantic features. The disclosed ASR system is jointly optimized on both speech segment end detection and ASR tasks. However, existing long-utterance training data for ASR systems typically does not include true value labels corresponding to speech segment ends, making supervised training of speech segment end joint networks difficult. The implementations herein are intended to perform a speech segment end annotation routine for automatically inserting speech segment end true value labels into true value training documents based on model hesitation and word duration. The example speech segment end scheme applies a heuristic-based weak supervision method, in which speech segment end true value labels are automatically inserted into true value training documents by applying one or more rules.

[0023] Figure 11 is a schematic diagram of an example system 100 and a speech environment 102. In the speech environment 102, the way in which a user 104 interacts with a computing device (such as a user device 10) may be through voice input. The user device 10 (also generally referred to as the device 10) is configured to capture sound (e.g., streaming audio data) from one or more users 104 within the speech environment 102. Here, the streaming audio data may refer to the spoken words 106 of the user 104, which serve as audible queries, commands for the device 10, or audible communications captured by the device 10. The voice-enabled system of the device 10 may process the query or command by answering the query and / or causing one or more downstream applications to execute / fulfill the command. The voice-enabled system of the device 10 may also generate a transcription of the audible communication captured by the device 10.

[0024] The user device 10 may correspond to any computing device associated with the user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablet computers, laptop computers, etc.), computers, wearable devices (e.g., smart watches), smart appliances, in-vehicle infotainment systems, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12. The memory hardware 14 stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 also includes an audio system 16 having: an audio capture device (e.g., a microphone) 16, i.e., 16a, for capturing spoken speech 106 within the speech environment 102 and converting the spoken speech into electrical signals; and a speech output device (e.g., a speaker) 16, i.e., 16b, for transmitting audible audio signals (e.g., as output audio data from the device 10). Although in the illustrated example, the user device 10 implements a single audio capture device 16 a, the user device 10 may implement an array of audio capture devices 16 a without departing from the scope of the present disclosure, wherein one or more capture devices 16 a in the array may not physically reside on the user device 10 but be in communication with the audio system 16 .

[0025] In the system 100, an automatic speech recognition (ASR) system 118 includes an RNN-T model 200 that implements a speech segment end model 205 and resides on a user device 10 of a user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executed in a cloud computing environment) that communicates with the user device 10 via a network 40. The user device 10 and / or the remote computing device 60 also includes an audio subsystem 108 that is configured to receive an utterance 106 spoken by the user 104 and captured by the audio capture device 16a, and convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 118. In the example shown, the user speaks a corresponding utterance 106, and the audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., an acoustic frame) 110 for input to the ASR system 118. Thereafter, the RNN-T model 200: receives as input an acoustic frame 110 corresponding to the utterance 106; generates an indication of whether the corresponding output step corresponds to an end of a speech segment that can be used to segment the utterance 106 into speech segments; and generates / predicts as output a corresponding transcription 120 for each speech segment of the utterance 106. In the example shown, the RNN-T model 200 can perform streaming speech recognition while the user is speaking to generate partial transcriptions (e.g., streaming speech recognition results) 120a, 120b.

[0026] The user device 10 and / or the remote computing device 60 also executes a user interface generator 107 that is configured to present a representation of a transcription 120, 120a-n of the utterance 106 to the user 104 of the user device 10, and present one or more responses 121, 121a-n to queries and / or commands of the utterance 106. As described in more detail below, the user interface generator 107 can display the speech recognition results 120, 120a-n and the responses 121, 121a-n during different times or at different points in time. As shown, the user interface generator 107 can present the speech recognition results 120, 120a-n (e.g., transcriptions) and the responses 121, 121a-n to represent the interaction / dialogue between the user 104 and the interactive program or application (e.g., the digital assistant application 50).

[0027] In some configurations, the transcription 120 output from the ASR system 118 for each identified speech segment is processed, for example, by a natural language processing / understanding (NLP / NLU) module executing on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. The digital assistant application 50 may provide an appropriate response 121 after executing the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) may convert the transcription 120 and / or the response 121 into synthesized speech for audible output by the user device 10 and / or another device.

[0028] In the illustrated example, the user 104 interacts with a program or application (e.g., digital assistant application 50) of the user device 10 using the ASR system 118. For example, Figure 1 The user 104 is depicted communicating with the digital assistant application 50 and the digital assistant application 50 displays the interactive digital assistant interface 18 on the screen 19 of the user device 10 to depict a natural conversational interaction between the user 104 and the digital assistant application 50. In this example, the user 104 speaks a first speech segment 106a of the utterance 106 ("Dictate the following message:") and then speaks a second speech segment 106b of the utterance 106 ("Where can I order Thai food?").

[0029] When the ASR system 118 receives an acoustic frame 110 corresponding to an utterance 106 while the user 104 is speaking, the speech segment end model 205 generates an indication in each output step as to whether the output step corresponds to the end of a speech segment. The speech segment end indication generated by the speech segment end model 205 can be used to delimit the speech segments 106 of the utterance 106 so that the RNN-T model 200 can perform speech recognition separately for each speech segment 106. In the example shown, the speech segment end model 205 detects or determines the end of a speech segment between the first and second speech segments 106, and outputs a speech segment end indication indicating the end of the first speech segment 106a.

[0030] For each speech segment 106 of the utterance 106 indicated or identified by the speech segment end model 205, the RNN-T model 200 performs speech recognition on the acoustic frames 110 corresponding to the speech segment 106 to produce a speech recognition result 120 corresponding to the speech segment 106. Continuing with the example, based on the speech segment end model 205 detecting the speech segment end between the first and second speech segments 106, during time 1, the user interface generator 107 presents a representation of the first speech recognition result 120a of the first speech segment 106a to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, so that words, word fragments and / or individual characters appear on the screen 19 of the user device 10 as soon as they are spoken.

[0031] Continuing with the example, based on the speech segment end model 205 detecting the speech segment end of the second speech segment 106, during time 2, the user interface generator 107 presents a representation of the second speech recognition result 120b of the second speech segment 106b to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, such that the words, word fragments and / or individual characters appear on the screen 19 of the user device 10 as soon as they are spoken. Then, during time 3, the user interface generator 107 presents a response 121b (i.e., “Do you want to send a message?”) via the digital assistant interface 18, which is generated by the assistant application 50 based on the speech recognition results 120a, 120b generated for the speech segments 106a, 106b.

[0032] Figure 2 2 is a schematic diagram of an example RNN-T model 200, which integrates a speech segment end model 205 as a first joint network 210 for detecting boundaries between spoken speech segments. As shown, the RNN-T model 200 includes an encoder network 220, a prediction network 300, a second joint network 240 (i.e., a word segment joint network 240), and a final softmax output layer 250. The encoder network 220 (e.g., an audio encoder), which is roughly similar to an acoustic model (AM) in a traditional ASR system, receives a sequence of feature vectors x = (x1, x2, ..., x t )222, of which (For example, Figure 1 The acoustic frame 110) and generates in each output step a representation of A high-order feature representation 224 (also commonly referred to as an acoustic representation) of .

[0033] In the example shown, the prediction network 300 comprises an LSTM-based prediction network that transforms the sequence y0, ..., y0 of non-whitespace symbols outputted so far by the softmax layer 250 like a language model (LM). u-1 252 is processed as hidden representation 350 (also commonly referred to as dense representation or language representation), the hidden representation represents the probability distribution of whether the current output step corresponds to the pause and end of the speech segment, where y0 represents a special sequence start symbol.

[0034] Figure 3 A prediction network 300 for the RNN-T model 200 is shown, which receives a sequence y of non-blank symbols. u-n ,...,y u-1 As input, the sequence of non-blank symbols is limited to the N previous non-blank symbols 252a-n output by the last softmax layer 250. In some examples, N is equal to two. In other examples, N is equal to five, however, the present disclosure is non-limiting and N can be equal to any integer. The sequence of non-blank symbols 252a-n can indicate a partial speech recognition result 120a, 120b ( Figure 1 ). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A-302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, the multi-head attention mechanism 302 can employ any number of heads. Notably, the multi-head attention mechanism 302 significantly improves performance with minimal increase in model size. As described in more detail below, each head 302A-H includes its own row of position vectors 308, and rather than causing an increase in model size by concatenating the outputs 318A-H from all heads, the outputs 318A-H are averaged by the head averaging module 322.

[0035] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses the shared embedding matrix 304 to represent the sequence y of non-blank symbols received as input in a corresponding output step from the plurality of output steps. ui-n ,...,y ui-1 Each non-whitespace symbol 301 in generates a corresponding embedding 306, namely 306a-n (e.g., ). It is worth noting that because the shared embedding matrix 304 is shared across all heads of the multi-head attention mechanism 302, the other heads 302B-H all generate the same corresponding embedding 306 for each non-blank symbol. Head 302A also adds a sequence of non-blank symbols y u-n ,...,y u-1 Each corresponding non-blank symbol in is assigned a corresponding position vector PV Aa–An 308, i.e. 308Aa-An (e.g. ). The corresponding position vector PV 308 assigned to each non-blank symbol indicates the position of the sequence of non-blank symbols (e.g., the N previous non-blank symbols output by the last Softmax layer 250) in the history. For example, the first position vector PV is assigned to the most recent position in the history. Aa , and the last position vector PV is assigned to the last position in the history of the N previous non-blank symbols output by the last Softmax layer 250 An Notably, each of embeddings 306 may include the same number of dimensions (ie, dimensional size) as each of position vectors PV 308 .

[0036] Although the sequence 252a-ny of non-blank symbols is represented by the shared embedding matrix 304 u-n ,...,y u-1 The corresponding embedding generated for each non-whitespace symbol 301 in is the same at all heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines a row PV of position vectors Aa-An , i.e., 308Aa-An, the second header 302B defines different rows of position vectors PV Ba-Bn , that is, 308 Ba-Bn , ..., and the Hth header 302H defines another different row PV of position vectors Ha-Hn , that is, 308 Ha-Hn .

[0037] For each non-blank symbol in the received sequence 252a-n of non-blank symbols, the first head 302A also weights the corresponding embedding 306 via the weight layer 310 in proportion to the similarity between the corresponding embedding and the corresponding position vector PV 308 assigned thereto. In some examples, the similarity may include a cosine similarity (e.g., a cosine distance). In the illustrated example, the weight layer 310 outputs a sequence 312Aa-An of weighted embeddings 312 each associated with the corresponding embedding 306, the corresponding embedding being weighted in proportion to the corresponding position vector PV 308 assigned thereto. In other words, the weighted embedding 312 output by the weight layer 310 for each embedding 306 may correspond to the dot product between the embedding 306 and the corresponding position vector PV 308. The weighted embeddings 312 may be interpreted as focusing on the embeddings in proportion to their similarity to the position associated with their corresponding position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, and therefore the sequence of weighted embeddings 312Aa-An is not concatenated, but averaged by a weighted average module 316 to generate a weighted average 318A of the weighted embeddings 312Aa-An as output from the first head 302A, which is represented by:

[0038]

[0039] In equation (1), h represents the index of head 302, n represents the position in the context, and e represents the embedding dimension. In addition, in equation (1), H and N include the sizes of the corresponding dimensions. Position vector PV 308 does not have to be trainable and may include random values. It is worth noting that even if weighted embedding 312 is averaged, position vector PV 308 can still potentially save position history information, thereby alleviating the need to provide recurrent connections at each layer of prediction network 300.

[0040] The operations described above with respect to the first head 302A are similarly performed by each other head 302B-H of the multi-head attention mechanism 302. Due to the different sets of positioning vectors PV 308 defined by each head 302, the weight layer 310 outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn at each other head 302B-H that is different from the sequence of weighted embeddings 312Aa-Aa at the first head 302A. Thereafter, the weighted average module 316 generates a corresponding weighted average 318B-H of the corresponding weighted embeddings 312 in the sequence of non-whitespace symbols as output from each other corresponding head 302B-H.

[0041] In the example shown, the prediction network 300 includes a head average module 322 that averages the weighted averages 318A-H output from the corresponding heads 302A-H. A projection layer 326 with SWISH may receive as input an output 324 corresponding to the average of the weighted averages 318A-H from the head average module 322 and generate as output a projection output 328. A final layer normalization 330 may normalize the projection output 328 to provide a single embedding vector P at a corresponding output step from the plurality of output steps. u 350 (i.e., hidden representation). The prediction network 300 generates only a single embedding vector P in each of the multiple output steps after the initial output step. u 350.

[0042] In some configurations, the prediction network 300 does not implement the multi-head attention mechanism 302, but only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through a projection layer 326 and a layer normalization 330 to provide a single embedding vector P u 350.

[0043] Return to reference Figure 2 , the speech segment ends and the joint network 210 (ie, the first joint network 210) receives a single embedding vector P from the prediction network 300u 350, and receiving a high-level feature representation from encoder 220 The speech segment end association network 210 generates a speech segment end indication (e.g., <eos>The conditional probability can be determined using the following mathematical expression:

[0044]

[0045] where x i is the i-th audio frame, and y i is the i-th decoded word in the bundle. When, for example, the conditional probability When a predefined threshold is met (eg, exceeded), the speech segment ends and the associated network 210 issues a speech segment end indication (eg, <eos>In some implementations, the speech segment end association network 210 is structurally identical to the word segment association network 240 and contains all word segments with the same logit value. In some examples, the training system 130 (see Figure 1 ) During the first phase, the encoder network 220, the prediction network 300, the word segment association network 240, and the softmax 250 are trained with a speech recognition task (e.g., using paired audio-transcription training data) to perform speech recognition. The paired audio-transcription training data includes audio data representing one or more spoken utterances and the corresponding transcriptions of the spoken utterances. Then, during the second phase, the training system 130 initializes the speech segment end association network 210 with the trained weights of the word segment association network 240 from the first phase, and uses the real value speech segment end indication (e.g., <eos>The speech segment end association network 210 is fine-tuned based on the word unit (word unit) to maximize the probability of emitting a true value speech segment end indicator. In some implementations, the training system 130 uses the FastEmit regularization term to train the speech segment end association network 210 to encourage the speech segment end association network 210 to emit as soon as sufficient context is available. <eos>The word unit 212 can more quickly identify multiple consecutive non-speech acoustic frames 110 in a sequence of acoustic frames 110 that meet the threshold duration.

[0046] In some implementations, the training data generator 140 (see Figure 1 ) Use a heuristic-based weak supervision method to convert the true value <eos>Tokens are automatically inserted into paired audio-transcribed training data (e.g., without any human annotation), where the ground truth <eos>Tokens are automatically inserted based on one or more rules. Example rules include, but are not limited to, inserting a real value end of speech segment at the end of the corresponding transcription. <eos>word units and end the true value speech segment at a position aligned with a non-speech segment (e.g., silence) of the audio data <eos>A word-mem is inserted into the corresponding transcription with the non-speech segment having a duration that satisfies a threshold duration, unless the non-speech segment of the audio data follows a word in the spoken utterance that is identified as a lengthening word (e.g., "heyyy") or the non-speech segment of the audio data follows a word in the spoken utterance that is identified as a filler word (e.g., "um"). Such exceptions reflect situations where the speaker may not have completed a speech segment. A lengthening word can be detected when, for example, the phoneme duration satisfies a standard deviation threshold (e.g., the phoneme duration exceeds five times the standard deviation of the phoneme duration). A filler word detection model can be used to detect filler words. In some implementations, the training data generator 140 applies rules and exceptions by obtaining silence, word, and phoneme durations using a forced alignment model applied to all audio-transcription pairs in the training data set.

[0047] The word fragment joint network 240 (ie, the second joint network 240) receives a single embedding vector P from the prediction network 300. u 350, and receiving a high-level feature representation from encoder 220 The word segment association network 240 generates a probability distribution of possible speech recognition hypotheses in the corresponding output step 242. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, e.g., one label for each of the 26 letters of the English alphabet and one label for a space. Thus, the word fragment association network 240 may output a set of values ​​indicating the likelihood of each of a predetermined set of output labels occurring. The set of values ​​may be a vector and may indicate a probability distribution for the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the set of output labels is not limited to this. For example, the set of output labels may include word fragments and / or entire words in addition to or in lieu of graphemes. The output distribution of the word fragment association network 240 may include a posterior probability value for each of the different output labels. Thus, when there are 100 different output labels representing different graphemes or other symbols, the output of the word fragment association network 240 may be a probability distribution of the number of possible occurrences of the word fragment association network 240. 242 can include 100 different probability values, one for each output label. Probability distribution of possible speech recognition hypotheses Indicates the speech recognition result 120 ( Figure 1 ). That is, the joint network 240 uses a single embedding vector 350 based on the sequence of non-whitespace symbols 252 to determine the probability distribution of the speech recognition result 120. In other words, the word segment joint network 240 generates a probability distribution 242 of possible speech recognition hypotheses at each output step (e.g., output step). The probability distribution can then be used 242 to select candidate orthographic elements (eg, graphemes, word fragments, and / or words) and assign scores thereto in a beam search process (eg, via a softmax layer 250 ) to determine the transcription 120 .

[0048] When the speech segment ends, the joint network 210 provides an indication of the end of the speech segment (e.g., <eos>When the word) 212, the softmax layer 250 can use any technique to select the distribution The output label / symbol with the highest probability in 242 is used as the next output symbol y predicted by the RNN-T model 200 in the corresponding output step. u 252. Thus, the RNN-T model 200 makes no conditional independence assumptions. Instead, the RNN-T model 200 predicts each symbol based not only on acoustics but also on the sequence of labels output so far. The RNN-T model 200 does assume that the output symbol is independent of future acoustic frames 110, which allows the RNN-T model 200 to be employed in a streaming manner. In some examples, the Softmax layer 250 is composed of a uniform set of word fragments or graphemes that are generated using all unique word fragments or graphemes in multiple training datasets.

[0049] In some implementations, the feature vector x222 input to the encoder network 220 may include an 80-dimensional logarithmic Mel filter bank feature formed by stacking three 25 millisecond (ms) acoustic frames with a 10 ms offset and downsampling to a frame rate of 30 ms. In some examples, the encoder network 220 includes twelve 512-dimensional conformer layers. The conformer layer includes a causal convolution with a kernel size of 15, and a stack of the above attention layers with 8 self-attention heads. Although the encoder network 220 described has a stack of multi-head attention layers / blocks with self-attention including a conformer layer / block (e.g., twelve conformer blocks), the present disclosure is not limited thereto. For example, the encoder network 220 may include a stack of transformer layers / or any other type of multi-head attention layer / block stack. The encoder network 220 may include a series of multi-head self-attention layers, deep convolutional layers, and feedforward layers. Alternatively, the encoder network 220 may include multiple long short-term memory (LSTM) layers instead of multi-head attention layers / blocks.

[0050] The prediction network 300 may include an LTSM-based network with an embedding dimension of 320. Dimension D of 232 joint Can be set to 640. In some examples, the word fragment association network 240 includes a hidden unit. Additionally or alternatively, the word fragment association network 240 does not include a fully connected (FC) layer. Alternatively, the prediction network 300 may include a stack of transformer or conformer blocks (or other types of multi-head attention blocks). The prediction network 300 can also be an embedding lookup table (e.g., a V2 embedding lookup table) to improve latency by outputting the sparse embedding looked up instead of generating a hidden representation. In some implementations, the prediction network 300 is a stateless prediction network.

[0051] The speech segment end joint network 210, the word segment joint network 240 and the prediction network 300 can jointly form the RNN-T decoder 260 of the RNN-T model 200. In some implementations, in order to further reduce the size of the RNN-T decoder 260 (i.e., the prediction network 300 and the word segment joint network 240), parameter binding between the prediction network 300 and the word segment joint network 240 is applied. Specifically, for the vocabulary size |V| and the embedding dimension d e , the shared embedding matrix 304 at the prediction network is Meanwhile, at the joint network 240, the last hidden layer includes the dimension size d h , the feed-forward projection weights from the hidden layer to the output logits will be There are extra blank tokens in the vocabulary. Therefore, the feed-forward layer corresponding to the last layer of the word segment association network 240 includes a weight matrix [d h ,|V]|. By letting the prediction network 300 embed the dimension d e The size of the word fragment association network 240 is the dimension d of the last hidden layer h The feed-forward projection weights of the word segment association network 240 and the shared embedding matrix 304 of the prediction network 300 can share their weights for all non-blank symbols via a simple transpose transformation. Since the two matrices share all their values, the RNN-T decoder 260 only needs to store these values ​​once in memory instead of storing two separate matrices. By placing the embedding dimension d e The size of is set to be equal to the hidden layer dimension d h The size of the RNN-T decoder 260 is reduced to equal the embedding dimension d e The number of parameters is the product of the vocabulary size |V|. This weight tying corresponds to the regularization technique.

[0052] In some implementations, the RNN-T model 200 performs a frame-synchronous beam search with a beam size of 8 and a pruning threshold of five (i.e., removing partial hypotheses whose negative log posterior exceeds the negative log posterior of the best hypothesis by 5). At each frame, the RNN-T model 200 performs a breadth-first search to find possible extensions, ignoring any extensions whose negative log posterior is five or greater, and limiting the search depth to ten extensions. In some examples, the RNN-T model 200 forces the end of speech segments that exceed sixty-five seconds in duration.

[0053] Figure 4 4 is a flow chart of an exemplary arrangement of operations of a computer-implemented method 400 for performing joint segmentation and automatic speech recognition. At operation 402, the method 400 includes receiving a sequence of acoustic frames (e.g., feature vectors x=(x1, x2, ..., x3) representing one or more utterances (e.g., utterance 106) t )110,222).

[0054] In each of the plurality of output steps, the method 400 performs operations 404, 406, and 408. At operation 404, the method 400 includes generating, by the encoder network 220 of the joint segmentation and automatic speech recognition (ASR) model (e.g., RNN-T 200), a high-level feature representation for a corresponding acoustic frame in the sequence of acoustic frames for each of the plurality of output steps. 224.

[0055] At operation 406, method 400 includes generating, by the decoder 260 of the joint segmentation and ASR model, a probability distribution of possible speech recognition hypotheses. 242. At operation 408, the method 400 includes generating, by the decoder 260 of the joint segmentation and ASR model, an indication of whether the corresponding output step corresponds to an end of a speech segment. Here, the joint segmentation and ASR model 200 is trained with a set of training samples, each training sample comprising: audio data representing a spoken utterance; and a corresponding transcription of the spoken utterance, the corresponding transcription having an end-of-speech segment ground-value token automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training sample.

[0056] Figure 5 5 is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computer devices. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0057] The computing device 500 includes a processor 510 (i.e., data processing hardware) that can be used to implement the data processing hardware 12 and / or 62, a memory 520 (i.e., memory hardware) that can be used to implement the memory hardware 14 and / or 64, a storage device 530 (i.e., memory hardware) that can be used to implement the memory hardware 14 and / or 64, a high-speed interface / controller 540 connected to the memory 520 and the high-speed expansion port 550, and a low-speed interface / controller 570 connected to the low-speed bus 560 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 580 coupled to the high-speed interface 540). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories may be used as appropriate. In addition, multiple computing devices 500 may be connected, each of which provides a portion of the necessary operations (e.g., as a server group, a blade server group, or a multi-processor system).

[0058] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0059] Storage device 530 is capable of providing mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various different implementations, storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0060] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 570 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 570 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 (which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)) can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0061] As shown, computing device 500 can be implemented in a variety of different forms. For example, computing device can be implemented as a standard server 500a or multiple implementations in a group of such servers 500a, implemented as a laptop computer 500b, or implemented as part of a rack server system 500c.

[0062] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpreted on a programmable system including at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0063] A software application (i.e., software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0064] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0065] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by special logic circuits (e.g., FPGA (field programmable gate array), graphics processing unit (GPU), tensor processing unit (TPU) or ASIC (application-specific integrated circuit)). Processors suitable for executing computer programs include, for example, both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices, or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0066] Unless expressly stated to the contrary, the phrase "at least one of A, B, or C" is intended to refer to any combination or subset of A, B, and C, such as: (1) only at least one A; (2) only at least one B; (3) only at least one C; (4) at least one A and at least one B; (5) at least one A and at least one C; (6) at least one B and at least one C; and (7) at least one A and at least one B and at least one C. Furthermore, unless expressly stated to the contrary, the phrase "at least one of A, B, and C" is intended to refer to any combination or subset of A, B, and C, such as: (1) only at least one A; (2) only at least one B; (3) only at least one C; (4) at least one A and at least one B; (5) at least one A and at least one C; (6) at least one B and at least one C; and (7) at least one A and at least one B and at least one C. Furthermore, unless expressly stated to the contrary, "A or B" is intended to refer to any combination of A and B, such as: (1) only A; (2) only B; and (3) A and B.

[0067] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and possibly a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0068] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.< / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. A joint segmentation and automatic speech recognition (ASR) model (200), characterized in that: include: An encoder (220), the encoder being configured to: receiving as input a sequence of acoustic frames (110) representing one or more utterances (106); and In each of the plurality of output steps, a high-level feature representation (224) is generated for a corresponding acoustic frame (110) in the sequence of acoustic frames (110); and A decoder (260), the decoder being configured to: receiving as input the high-level feature representation (224) generated by the encoder (220) in each of the plurality of output steps; and In each of the plurality of output steps, generating: a probability distribution of possible speech recognition hypotheses (242); and an indication (212) of whether the corresponding output step corresponds to an end of a speech segment, wherein the joint segmentation and ASR model (200) is trained using a set of training samples, each training sample in the set of training samples comprising: Audio data representing spoken utterances; as well as A corresponding transcription of the spoken utterance, the corresponding transcription having an end-of-segment-of-speech ground-value token automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training samples.

2. The joint segmentation and ASR model (200) of claim 1, wherein the decoder (260) comprises: A prediction network (300) configured to, in each of the plurality of output steps: receiving as input a sequence of non-whitespace symbols (252) output by the last softmax layer (250); and generating a hidden representation (350); A first joint network (210), wherein the first joint network is configured to: receiving as input the hidden representation (350) generated by the prediction network (300) in each of the plurality of output steps and the high-level feature representation (224) generated by the encoder (220) in each of the plurality of output steps; and generating, in each of said plurality of output steps, said indication as to whether said corresponding output step corresponds to an end of a speech segment (212); and A second joint network (240), the second joint network being configured to: receiving as input the hidden representation (350) generated by the prediction network (300) in each of the plurality of output steps and the high-level feature representation (224) generated by the encoder (220) in each of the plurality of output steps; and In each of the plurality of output steps, the probability distribution of possible speech recognition hypotheses is generated (242).

3. The joint segmentation and ASR model (200) of claim 2, wherein: In each of the plurality of output steps: The sequence of previous non-whitespace symbols (252) received as input at the prediction network (300) comprises a sequence of N previous non-whitespace symbols (252) output by the last softmax layer (250); and The prediction network (300) is configured to generate the hidden representation (350) by: For each non-whitespace symbol (252) in the sequence of N previous non-whitespace symbols (252), generating a corresponding embedding (306); and An average embedding (318) is generated by averaging the corresponding embeddings (306), the average embedding (318) comprising the hidden representation (350).

4. The joint segmentation and ASR model (200) of claim 2, wherein the prediction network (300) comprises a V2 embedding lookup table.

5. The joint segmentation and ASR model (200) of any one of claims 1 to 4, wherein a training process (130) trains the joint segmentation and ASR model (200) with the set of training samples by: During a first phase, the second joint network (240) is trained to learn how to predict the corresponding transcription of the spoken utterance (106) represented by the audio data of each training sample; and After training the second joint network (240), during the second phase: Initializing the first joint network (210) using the same parameters as the trained second joint network (240); and The speech segment-ending ground-value tokens inserted into the corresponding transcription of the spoken utterance represented by the audio data of each training sample are used.

6. The joint segmentation and ASR model (200) as claimed in any one of claims 1 to 5, wherein the encoder (220) comprises a causal encoder (220), the causal encoder comprising a stack of conformer layers or transformer layers.

7. The joint segmentation and ASR model (200) as claimed in any one of claims 1 to 6, wherein the speech segment end real value word is automatically inserted into the corresponding transcription without any manual annotation.

8. The joint segmentation and ASR model (200) of any one of claims 1 to 7, wherein the set of heuristic-based rules and exceptions applied to each training sample in the set of training samples comprises: inserting the real-value speech segment end token at the end of the corresponding transcription; as well as inserting the real-value end-of-speech segment token into the corresponding transcription at a location aligned with a non-speech segment of the audio data, the non-speech segment having a duration that satisfies a threshold duration unless: The non-speech segment of the audio data follows a word in the spoken utterance (106) that is identified as a lengthened word; or The non-speech segment of the audio data follows a word in the spoken utterance (106) that is identified as a filler word.

9. The joint segmentation and ASR model (200) of claim 8, wherein the word in the spoken utterance is identified as the lengthened word when the phoneme duration of the word in the spoken utterance satisfies a standard deviation threshold.

10. The joint segmentation and ASR model (200) of claim 8, wherein: After training the joint segmentation and ASR model (200), the decoder (260) is configured to issue the indication (212) that the corresponding output step corresponds to the end of the speech segment earlier than a plurality of consecutive non-speech acoustic frames (110) in the sequence of identified acoustic frames (110) that meet the threshold duration.

11. The joint segmentation and ASR model (200) of any one of claims 1 to 10, wherein the joint segmentation and ASR model (200) is trained to maximize the probability of uttering a true value label of the end of speech segment.

12. A computer-implemented method (400) executed on data processing hardware (510), thereby causing the data processing hardware (510) to perform operations, characterized in that The operations include: receiving a sequence of acoustic frames (110) representing one or more utterances (106); and In each of the multiple output steps: generating, by an encoder (220) of a joint segmentation and automatic speech recognition (ASR) model (200), a high-level feature representation (224) for a corresponding acoustic frame in the sequence of acoustic frames (110); and Generated by the decoder (260) of the joint segmentation and ASR model (200): a probability distribution of possible speech recognition hypotheses (242); and an indication (212) of whether the corresponding output step corresponds to an end of a speech segment, wherein the joint segmentation and ASR model (200) is trained using a set of training samples, each training sample in the set of training samples comprising: audio data representing spoken utterances; and A corresponding transcription of the spoken utterance, the corresponding transcription having an end-of-segment-of-speech ground-value token automatically inserted into the corresponding transcription based on a set of heuristic-based rules and exceptions applied to the training samples.

13. The computer-implemented method (400) of claim 12, wherein: The operations also include generating a hidden representation (350) based on a sequence of non-whitespace symbols (252) output by the last softmax layer (250) using a prediction network (300) of the decoder (260) in each of the plurality of output steps; Generating the indication (212) as to whether the corresponding output step corresponds to the end of the speech segment comprises: using a first joint network (210) of the decoder (260), generating the indication (212) as to whether the corresponding output step corresponds to the end of the speech segment based on the hidden representation (350) generated by the prediction network (300) in each of the plurality of output steps and the high-order feature representation (224) generated by the encoder (220) in each of the plurality of output steps; and Generating the probability distribution (242) of possible speech recognition hypotheses includes: using a second joint network (240) of the decoder (260) to generate the probability distribution (242) of possible speech recognition hypotheses based on the hidden representation (350) generated by the prediction network (300) in each of the multiple output steps and the high-order feature representation (224) generated by the encoder (220) in each of the multiple output steps.

14. The computer-implemented method (400) of claim 13, wherein: The sequence of previous non-whitespace symbols (252) received as input at the prediction network (300) comprises a sequence of N previous non-whitespace symbols (252) output by the last softmax layer (250); and Generating the hidden representation (350) using the prediction network (300) includes generating the hidden representation (350) by: For each non-whitespace symbol (252) in the sequence of N previous non-whitespace symbols (252), generating a corresponding embedding (306); and An average embedding (318) is generated by averaging the corresponding embeddings (306), the average embedding (318) comprising the hidden representation (350).

15. The computer implemented method (400) of claim 13, wherein the prediction network (300) comprises a V2 embedding lookup table.

16. The computer-implemented method (400) of any one of claims 12 to 15, wherein a training process (130) trains the joint segmentation and ASR model (200) with the set of training samples by: During a first phase, the second joint network (240) is trained to learn how to predict the corresponding transcription of the spoken utterance (106) represented by the audio data of each training sample; and After training the second joint network (240), during the second phase: Initializing the first joint network (210) using the same parameters as the trained second joint network (240); and The speech segment-ending ground-value tokens inserted into the corresponding transcription of the spoken utterance represented by the audio data of each training sample are used.

17. A computer-implemented method (400) as claimed in any one of claims 12 to 16, wherein the encoder (220) comprises a causal encoder (220) comprising a stack of conformer layers or transformer layers.

18. The computer-implemented method (400) of any one of claims 12 to 17, wherein the end-of-speech-segment real-value word-gram is automatically inserted into the corresponding transcription without any manual annotation.

19. The computer-implemented method (400) of any one of claims 12 to 18, wherein the set of heuristic-based rules and exceptions applied to each training sample in the set of training samples comprises: inserting the real-value speech segment end token at the end of the corresponding transcription; as well as inserting the real-value end-of-speech segment token into the corresponding transcription at a location aligned with a non-speech segment of the audio data, the non-speech segment having a duration that satisfies a threshold duration unless: The non-speech segment of the audio data follows a word in the spoken utterance (106) that is identified as a lengthened word; or The non-speech segment of the audio data follows a word in the spoken utterance (106) that is identified as a filler word.

20. The computer-implemented method (400) of claim 19, wherein the word in the spoken utterance (106) is identified as the lengthened word when a phoneme duration of the word in the spoken utterance (106) satisfies a standard deviation threshold.

21. The computer-implemented method (400) of claim 19, wherein: After training the joint segmentation and ASR model (200), the operation further includes: for a plurality of consecutive non-speech acoustic frames (110) satisfying the threshold duration in the sequence earlier than the identified acoustic frame (110), the decoder (260) issues the indication (212) that the corresponding output step corresponds to the end of the speech segment.

22. The computer-implemented method (400) of any one of claims 12 to 21, wherein the joint segmentation and ASR model (200) is trained to maximize the probability of uttering a true value label of the end of speech segment.

Images