Regularizing word segmentation

Abstract

A method (600) for subword,segmentation includes receiving an input word (302) to be segmented into a plurality of subword units (119).,The method further includes executing a subword segmentation routine (300) to segment the,input word into a plurality of subword units by accessing a,trained vocabulary set (350) of subword units, and selecting a,plurality of subword units from the input word by greedily finding the,longest subword unit from the input word that is present in the,trained vocabulary set until the end of the input word is,reached.

Description

[Technical Field]

[0001] TECHNICAL FIELD This disclosure relates to regularizing word segmentation. [Background technology]

[0002] An automatic speech recognition (ASR) system evolves from multiple models (e.g., acoustic, pronunciation, and language models), each with its own purpose, into an integrated model in which a single neural network is used to directly map an audio waveform (i.e., an input sequence) to an output sentence (i.e., an output sequence). This integration results in a sequence-to-sequence approach, which generates a sequence of words or graphemes given a sequence of audio features. With this integrated structure, all components of the model can be trained together as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is composed entirely of neural networks. A complete neural network functions without the inclusion of external and / or manually designed components (e.g., finite state transducers, lexicons, or text normalization modules). Furthermore, when training E2E models, these models generally do not require bootstrapping from a decision tree or time alignment from another system. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a computer-implemented method for subword segmentation, which, when executed on data processing hardware, causes the data processing hardware to perform operations including receiving an input word to be segmented into a plurality of subword units, and also includes executing a subword segmentation routine to segment the input word into a plurality of subword units by the substeps of accessing a trained vocabulary set of subword units and selecting a plurality of subword units from the input word by greedily finding the longest subword unit from the input word that is present in the trained vocabulary set, until the end of the input word is reached.

[0004] Embodiments of the present disclosure may include one or more of the following optional features: In some embodiments, the substep of selecting a plurality of subword units includes, for each corresponding position of the plurality of different positions of the input word, identifying all possible candidate subword units from the input word at the corresponding position that are present in the trained vocabulary set, and randomly sampling from all possible candidate subword units by assigning a probability of 1-p to the longest of the possible candidate subword units and dividing the remaining p probabilities evenly among all possible candidate subword units from the input word at the corresponding position. The operations may further include, prior to the step of executing the subword segmentation model, creating typos for the input word by randomly removing letters from the input word, individually, with pre-specified probabilities.

[0005] In some examples, the operations also include, prior to executing the subword segmentation model, creating typos for the input word by pre-specifying probabilities for swapping the order of adjacent character pairs, and for each adjacent character pair in the input word, swapping the order of characters from adjacent character pairs in the input word based on the pre-specified probabilities, wherein any given character order in the input word is swapped at most once.

[0006] In some embodiments, the operations also include receiving training examples including audio data characterizing the utterance of the input word, and processing the audio data to generate speech recognition results for the utterance of the input word for output by the speech recognition model. Here, the speech recognition results include a set of hypothesized subword units, each output from the speech recognition model in a corresponding output step. In these embodiments, the operations further include determining a supervised loss term based on the set of hypothesized subword units and a plurality of subword units selected from the input word by the subword segmentation routine, and updating parameters of the speech recognition model based on the supervised loss term. In some examples, the speech recognition model includes a recurrent neural network transformer (RNN-T) model architecture having an audio encoder, a predictive network, and a joint network. Here, the audio encoder can include one of multiple long short-term memory (LSTM) layers, multiple transformer layers, or multiple conformer layers. In another example, the speech recognition model includes a listen, attend, spell (LAS) model architecture having an audio encoder, an attention mechanism, and a decoder.

[0007] Another aspect of the present disclosure provides a system for subword segmentation, the system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving an input word to be segmented into a plurality of subword units. The operations also include executing a subword segmentation routine to segment the input word into a plurality of subword units by the substeps of accessing a trained vocabulary set of subword units and selecting a plurality of subword units from the input word by greedily finding the longest subword unit from the input word that is present in the trained vocabulary set until the end of the input word is reached.

[0008] Embodiments of the present disclosure may include one or more of the following optional features: In some embodiments, the substep of selecting a plurality of subword units includes, for each corresponding position of a plurality of different positions of the input word, identifying all possible candidate subword units from the input word at the corresponding position that are present in the trained vocabulary set, and randomly sampling from all of the possible candidate subword units by assigning a probability of 1-p to the longest of the possible candidate subword units and dividing the remainder of the p probabilities evenly among all of the possible candidate subword units from the input word at the corresponding position. The operations may further include, prior to the step of executing the subword segmentation model, creating typos for the input word by randomly removing letters from the input word, individually, with pre-specified probabilities.

[0009] In some examples, the operations also include, prior to executing the subword segmentation model, creating typos for the input word by pre-specifying probabilities for swapping the order of adjacent character pairs, and for each adjacent character pair in the input word, swapping the order of characters from adjacent character pairs in the input word based on the pre-specified probability, wherein any given character order in the input word is limited to being swapped at most once.

[0010] In some embodiments, the operations also include receiving training examples including audio data characterizing the utterance of the input word and processing the audio data to generate speech recognition results for the utterance of the input word for output by the speech recognition model. Here, the speech recognition results include a set of hypothesized subword units, each output from the speech recognition model in a corresponding output step. In these embodiments, the operations further include determining a supervised loss term based on the set of hypothesized subword units and a plurality of subword units selected from the input word by the subword segmentation routine, and updating parameters of the speech recognition model based on the supervised loss term. In some examples, the speech recognition model includes a recurrent neural network transformer (RNN-T) model architecture having an audio encoder, a prediction network, and a joint network. Here, the audio encoder can include one of multiple long short-term memory (LSTM) layers, multiple transformer layers, or multiple conformer layers. In other examples, the speech recognition model includes a listen, attend, spell (LAS) model architecture having an audio encoder, an attention mechanism, and a decoder.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an example system for subword segmentation for training a speech recognition model. [Figure 2A] FIG. 1 is a schematic diagram of an exemplary recurrent neural network transformer (RNN-T) model architecture. [Figure 2B] FIG. 1 is a schematic diagram of an exemplary listen, attend, and spell model architecture. [Figure 3] 1 is an example of a subword segmentation routine that segments an input word into multiple subword segments. [Figure 4] 4 is an exemplary algorithm illustrating the subword segmentation routine of FIG. 3. [Figure 5] FIG. 2 is a schematic diagram of an exemplary subgraph representing candidate subword units sampled from all possible subword units; [Figure 6] 1 is a flowchart of an exemplary arrangement of operations for a computer-implemented method of segmenting subword units. [Figure 7] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0013] Like reference symbols in the various drawings indicate like elements.

[0014] Current automatic speech recognition (ASR) systems focus on providing not only high quality (e.g., low word error rate (WER)) but also low latency (e.g., short delay between the user speaking and the appearing transcription). The development of an end-to-end sequence ASR model provides a single neural network model to directly receive audio representing speech as input and output recognized text, without using separate, separately trained acoustic, linguistic, and pronunciation models. With an end-to-end sequence ASR model, word-based and grapheme-based representations of text are two simple techniques for representing the output recognized text. Generally, word-based representations result in text sequences with shorter length compared to grapheme-based representations, making it easier for the ASR model to learn cross-context dependencies. However, implementing an ASR model to use word-based representations requires a predefined vocabulary to recognize possible words, thereby limiting its ability to process out-of-vocabulary (OOV) terms / words that were not present in the training data. Another drawback of word-based representations is that without incorporating additional information into the generated word embeddings, ASR models cannot learn relationships between linguistically related words, such as "catching" versus "catch," "books" versus "book," or "helpful" versus "helpful."

[0015] On the other hand, grapheme-based representations involve outputting sequences of single characters that form words. While ASR models using grapheme-based representations can theoretically learn relationships between similarly spelled words and process OOV terms / words, their main drawback is that they require a larger memory footprint and require more computational time to generate much longer text sequences. In other words, creating a grapheme-based representation increases the number of decoding steps, which has the counterproductive effect of slowing down inference speed. Furthermore, grapheme-based representations hinder the ability to learn patterns from data, resulting in suboptimal performance.

[0016] Embodiments herein are directed to using an end-to-end sequence ASR model that directly maps input audio data (i.e., an input sequence) to an output sequence in the form of a subword-based text representation, including a series of subword units that form one or more words. The subword-based text representation achieves a trade-off between the two approaches of word-based and grapheme-based representation. Embodiments herein are further directed specifically to a subword segmentation routine (also referred to as a "word tokenization routine / algorithm" or simply a "word piece model") that improves the subword-based representation of text for training a sequence ASR model. As discussed in more detail below, executing the subword segmentation routine segments an input word into multiple subword units by accessing a trained vocabulary set of subword units and sequentially selecting subword units for inclusion in the multiple subword units by greedily finding the longest subword units from the input word that are present in the trained vocabulary set until the end of the input word is reached. As used herein, the term "subword unit" can be referred to interchangeably with "word piece" and can range from a grapheme to an entire word. Obviously, the vocabulary set of subword units accessed by the subword segmentation routine need not include any OOV terms / words; the subword segmentation routine only needs to be trained to maximize the likelihood of a language model for a training set of input words. The subword segmentation routine can be correlated to a statistical word piece model trained using word counts obtained from text data to segment each input word individually into subword units. Symbols indicating individual spaces can be included in the subword units.

[0017] The input word may include one of one or more words in a ground truth transcription of a training utterance characterized by training audio data. Thus, a plurality of subword units segmented from the input word and selected by the subword segmentation routine may serve as ground truth subword units used to train the ASR model to learn to predict an output sequence of subword units from input training audio data characterizing the training utterance. The audio data characterizing the training utterance may include a non-synthetic speech representation corresponding to the training utterance spoken by a human, or a synthesized speech representation converted from input text (e.g., a ground truth transcription) by text-to-speech synthesis.

[0018] Further embodiments are directed to applying one or more regularization techniques to input words prior to segmentation by the subword segmentation routine. As discussed in more detail below, these regularization techniques include modifying the spelling of input words prior to performing the subword segmentation routine, which can provide a sophisticated form of label smoothing to improve the performance of the ASR model.

[0019] 1 is an example of an illustrative system 100 for an audio environment. In an audio environment, a user 104 interacts with a computing device, such as a user device 10, through voice input. The user device 10 (also generally referred to as a device 10) is configured to capture sounds (e.g., streaming audio data) from one or more users 104 in the audio environment. As used herein, streaming audio data may refer to utterances 106 spoken by the user 104, which function as audible inquiries, commands to the device 10, or audible communications captured by the device 10. The conversation-enabled system of the device 10 can respond to the inquiries or commands by replying to the inquiries and / or causing one or more downstream applications to perform / accomplish the commands.

[0020] The user device 10 may correspond to any computing device associated with a user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches, smart headphones, headsets, etc.), smart appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 that communicates with the data processing hardware 12 and stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 comprising audio capture devices 16, 16a (e.g., microphones) for capturing and converting speech 106 spoken within an audio environment into electrical signals, and audio output devices 16, 16b (e.g., speakers) for communicating audible audio signals (e.g., as output audio data from the device 10). Although the user device 10 implements a single audio capturing device 16a in the illustrated example, the user device 10 may implement an array of audio capturing devices 16a without departing from the scope of the present disclosure, such that one or more capturing devices 16a in the array are not physically present on the user device 10 and can communicate with the audio system 16.

[0021] The system 100 includes an automatic speech recognition (ASR) system 118 that implements an end-to-end sequential ASR model 200 trained to directly map input audio data 110 corresponding to the utterance 106 to a subword-based text representation 119 that includes a series of subword units 119 that form words in a transcription 120 of the utterance 106. The ASR system 118 executes a subword segmentation routine 300 (also referred to as a “word tokenization routine / algorithm” or simply a “wordpiece model”) that improves the subword-based text representation 119 for training the sequential ASR model 200. As discussed in more detail below with reference to FIG. 3 , the ASR system 118 executes the subword segmentation routine 300 to segment an input word 302 ( FIG. 3 ) into multiple subword units 119, in the form of the subword-based text representation 119, which is used to improve the speech recognition accuracy of the ASR model 200 that is trained to predict an output label.

[0022] Clearly, the subword units 119 segmented by the subword segmentation routine 300 better reflect the speech of people with accents, diverse pronunciations, and / or fluency deficits. Therefore, the subword segmentation routine 300 complements the multilingual ASR model 200, which can recognize speech spoken in various languages. For example, the multilingual ASR model 200 can be trained to recognize speech from multilingual speakers, such as speakers who switch between different Indian languages, whereby the subword units 119 segmented by the subword segmentation routine 300 improve the performance of the ASR model 200 when these multilingual speakers switch between languages ​​and change the pronunciation of a word and its neighboring words. By the same concept, the subword units 119 segmented by the subword segmentation routine 300 can enable the ASR model 200 (monolingual or multilingual) to include recognition of all types of irregular / accented speech that may be spoken by different speakers.

[0023] The ASR system 118 may reside on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) that is in communication with the user device 10 via the network 40. The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 and captured by the audio capturing device 16a and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 (e.g., audio data) that can be processed by the ASR system 118. In the illustrated example, the user speaks each utterance 106, and the audio subsystem 108 converts the utterance 106 into corresponding audio data 110 (e.g., acoustic frames) as input to the ASR system 118. Thereafter, the ASR model 200 receives as input audio data 110 corresponding to the utterance 106 and generates / predicts corresponding sub-word units as output in each of a plurality of output steps to form a transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106. In the illustrated example, the ASR model 200 can perform streaming speech recognition to generate an initial speech recognition result 120 that can be rewritten to generate a final speech recognition result 120.

[0024] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of a transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 can display initial speech recognition results 120 in a streaming manner, followed by a final speech recognition result 120. In some configurations, the transcription 120 output from the ASR system 118 is processed by, for example, a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) can convert the transcription into synthesized speech for audible output by the user device 10 and / or another device.

[0025] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) on a user device 10 that uses an ASR system 118. For example, FIG. 1 shows a user 104 communicating with a digital assistant application 50, which displays a digital assistant interface 18 on the screen of the user device 10 to show a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is a spoken utterance 106 captured by an audio capture device 16a and processed by an audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts it into acoustic frames 110 for input to the ASR system 118.

[0026] In the example shown in FIG. 1 , digital assistant application 50 can use natural language processing to respond to a question posed by user 104. Natural language processing generally refers to the process of interpreting written language (e.g., initial speech recognition result 120a and / or final speech recognition result 120b) and determining whether the written language prompts an action. In this example, digital assistant application 50 uses natural language processing to recognize that the question from user 104 is about the user's schedule, and more specifically, about a concert on the user's schedule. By recognizing these details using natural language processing, the automated assistant returns response 19 to the user's query, stating, "The venue doors open at 6:30 PM, and the concert starts at 8 PM." In some configurations, natural language processing occurs on a remote server 60 in communication with data processing hardware 12 of user device 10.

[0027] 2A , in some embodiments, the ASR model 200 includes a recurrent neural network transformer (RNN-T) model 200a architecture (or other type of frame alignment-based transformer model) that conforms to latency constraints associated with interactive applications. The RNN-T model 200a offers a small computational footprint and utilizes smaller memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 102 (e.g., without requiring communication with a remote server). The RNN-T model 200a includes an encoder network 210, a prediction network 220, and a joint network 230. The prediction network 220 and the joint network 230 can collectively provide an RNN-T decoder 231. The encoder network 210, roughly analogous to an acoustic model (AM) in conventional ASR systems, can include a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder may generate a sequence of d-dimensional feature vectors (e.g., acoustic frame 110 (FIG. 1)) x=(x1, x2, . . . , x T ), where x t

[0028]

number

[0029] and at each time step, generate a higher-level representation. This higher-level representation is

[0030]

number

[0031] is shown as:

[0032] Similarly, the prediction network 220 is also an LSTM network, which, like a language model (LM), currently calculates y 0 , . . . , yui-1 The sequence of space-free subword units output by the final softmax layer 240 is represented as

[0033]

number

[0034] 225. Obviously, the non-blank subword units (y0,...,y ui-1 ) sequence captures the linguistic dependencies between the non-blank symbols predicted during the previous time step and the blank symbols during the current time step to assist the joint network 230 in predicting the probability of the next output subword unit 219.

[0035] Finally, using an RNN-T model architecture, the representations generated by the encoder network 210 and the prediction network 220 are combined by the joint network 230. The joint network then:

[0036]

number

[0037] , which is the distribution for the next subword unit 119. In other words, at each output step (e.g., time step), the joint network 230 generates a probability distribution for the hypothesized subword unit 119. Here, a "hypothesized subword unit" corresponds to a possible subword unit from a trained vocabulary set 350 ( FIG. 3 ) of subword units, each of which represents a corresponding subword unit in a specified natural language. The subword units included in the vocabulary set 350 span individual graphemes, word pieces, and entire words. In some examples, the trained vocabulary set 350 includes 4096 different subword units in a specified natural language. In these examples, the joint network 230 is trained to predict / output 4096 different subword units. The output distribution of the joint network 230 can include a posterior probability value for each of the different hypothesized subword units. Thus, if there are 4096 different subword units representing various graphemes, word pieces, or words, the output y of the joint network 230 can be calculated as follows: i can contain 4096 different probability values, one for each output label. The probability distributions can then be used to select and assign scores to candidate subword units (e.g., graphemes, wordpieces, and / or words) in a beam search process (e.g., by softmax layer 240) to determine transcription 120.

[0038] The softmax layer 240 can select the output subword unit with the highest probability in the distribution using any technique, and the next subword unit is predicted by the RNN-T model 200a in the corresponding output step. In this way, the RNN-T model 200a does not make a conditional independent assumption, but rather the prediction of each subword unit is conditioned not only on the acoustics but also on the sequence of subword units output so far. The RNN-T model 200a assumes that the output subword units are independent of future acoustic frames 110, thereby enabling the RNN-T model to be used in a streaming manner.

[0039] In some examples, the encoder network 210 of the RNN-T model 200 is made up of eight 2048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. In other embodiments, the encoder network 210 includes a network of conformer or transformer layers. The prediction network 220 may have two 2048-dimensional LSTM layers, each also followed by a 640-dimensional projection layer and a 128-unit embedding layer. Finally, the joint network 230 may also have 640 hidden units. The softmax layer 240 may be composed of a unified set of word pieces / subword units generated using all unique word pieces / subword units in the training data.

[0040] 2B, in another embodiment, the ASR model 200 includes a listen, attend, and spell (LAS) model 200b architecture, which provides a single neural network including a listener-encoder model 211, which is similar to a traditional acoustic model, an attendant module 221, which acts as an alignment model, and a decoder 231, which is similar to a language model in traditional systems. In particular, the listener-encoder module 211 takes input features x (e.g., acoustic frame 110 (FIG. 1)) and converts them into a higher-level feature representation h encThe encoded feature representation h enc This process of generating f can be done for each of multiple input frames, representing different input time steps. These time steps are denoted by the subscript u below. Thus, for a set of frames {f, f, f, . . . , f u}, the corresponding set of coded outputs {h1, h2, h3, . . . , h u} may be present.

[0041] The output of the listener encoder module 211 is passed to an attendant module 221, which, like a dynamic time warping (DTW) alignment module, determines the next output sub-word unit y i To predict h enc In some examples, the attender module 221 is referred to herein as an attender neural network or attender 221. The attender 221 determines which encoder features in i Each context output vector c i For example, the attender 221 may calculate attention based on the encoding for one or more input steps u, e.g., based on the encoding for the current input step as well as the encoding for the previous input step. u For the set of all encoder outputs of an utterance, such as i The attention context vector may be a vector that represents a weighted summary of the current and previous encodings for a frame (e.g., part) of speech to be recognized.

[0042] Finally, the output of the attender 221 is passed to a decoder 231, which uses the attention context c output by the attender 221 to create the decoder output. i(e.g., context vector or attention distribution), as well as the previous prediction y i-1 The decoder output is the embedded subword unit {y i-1 , y0}, and the input x, the current hypothesized subword unit y i For the probability distribution P(y i |y i-1 , y0, x). Thus, at each output step, decoder 231 generates a probability distribution over hypothesized subword units 119. Similar to RNN-T model 200a discussed above with reference to FIG. 2A, the "hypothesized subword units" correspond to possible subword units from a trained vocabulary set of subword units 350 (FIG. 3), each of which expresses a corresponding subword unit in a specified natural language.

[0043] Although not shown, LAS model 200b may include a softmax layer that receives the output of decoder 231. In some embodiments, the softmax layer is separate from decoder 231 and receives the output y i , and the output of the softmax layer is then used in a beam search process to select the subword units that form the words of the resulting transcription 120. In some embodiments, the softmax layer is integrated with the decoder 231, so that the output y i represents the output of the softmax layer.

[0044] The decoder 231 and / or associated softmax layer may be trained to output a set of values ​​indicating the likelihood of occurrence of each of a given set of subword units 119. This set of values ​​may be a vector and may indicate a probability distribution over the set of subword units. The output distribution of the decoder 231 and / or softmax layer may include posterior probability values ​​for each of the various subword units. Thus, if there are 4096 different subword unit labels, the decoder output yi , or output y i The output of a softmax layer that receives and processes σ can include 4096 different probability values, one for each output label. The probability distributions can then be used to select and assign scores to candidate subword units 119 in a beam search process to determine transcription 120.

[0045] 3 illustrates an example of a subword segmentation routine 300 (e.g., lightweight WPM) that segments an input word 302 into multiple subword units 119, 119a-b. During execution of the subword segmentation routine 300, the routine 300 accesses a trained vocabulary set 350 of subword units. In some examples, the trained vocabulary set 350 includes 4,096 different subword units, including word pieces as well as graphemes and whole words. The trained vocabulary set 350 can be stored in a data storage device 360 ​​of the remote system 60. During execution of the subword segmentation routine 300, the routine 300 selects multiple subword units 219 from the input word 302 by greedily finding the longest subword unit present in the trained vocabulary set 350 from the input word 302, until it reaches the end of the input word 302. That is, all processing performed by routine 300 is performed strictly from left to right to sequentially find the longest subword present in trained vocabulary set 350 at each position / index. For example, if input word 302 contains the word "Interspeech," the longest subword unit selected at the first position / index would be "Inter," followed by "sp," "ee," and "ch," and so on. Figure 4 shows an exemplary algorithm 400 representing subword segmentation routine 300.

[0046] 3 also illustrates using multiple subword units 119, 119a-n as ground truth subword units for training ASR model 200. Here, training examples including audio data 110T characterizing the utterance of input word 302 may be paired with the ground truth subword units 119. ASR model 200 processes training audio data 110 to generate, as output, speech recognition results 201 for the utterance of input word 302. Here, speech recognition results 201 include a series of hypothesized subword units, where each hypothesized subword unit in the series of hypothesized subword units is output from speech recognition model 200 in a corresponding output step. Thereafter, loss function 380 (e.g., cross-entropy loss or RNN-T loss) determines supervised loss term 390 based on the series of hypothesized subword units and the multiple subword units 119 selected from the input word by subword segmentation routine 300. The supervised loss term 390 is used to update the parameters of the ASR model 200 during training.

[0047] In some embodiments, randomness is added to the subword unit selection step (e.g., line 4 of algorithm 400 in FIG. 4 ) by sampling the selection of subword units at each index from all possible subword units, rather than simply always selecting the longest subword unit at each index. That is, for each corresponding position (e.g., index) of multiple different positions in the input word, routine 300 can identify all possible candidate subword units from input word 302 at the corresponding position that exist in the trained vocabulary set. Thereafter, for the corresponding position in input word 302, routine 300 can then randomly sample from all of the possible candidate subword units by assigning a 1−p probability to the longest of the possible candidate subword units and dividing the remainder of the p probability evenly among all of the possible candidate subword units from input word 302 at the corresponding position. Continuing with the above example in which input word 302 is “Interspeech,” FIG. 5 shows a subgraph 500 of a training procedure for randomly sampling the selection of subword units at each position from all possible subword units. Here, the regularization rate is uniform and can include a value of 0.1, so that 0.9 probability is assigned to the longest candidate subword unit at each position, while the remaining 0.1 is divided evenly among all of the candidate subword units (including the longest candidate). Obviously, multiple subword unit candidates sharing the same prefix are sampled at different positions, which can have two effects: the true distribution of always selecting the longest subword unit is not sharp, thus preventing the model from being overconfident, and the model becomes aware of the prefix relationships of the subword units.

[0048] Referring back to FIG. 3 , in some embodiments, one or more regularization techniques 305 are applied prior to executing the subword segmentation routine 300 to introduce randomness into the routine 300 to further enhance gains in speech recognition performance by the ASR model 200. In some examples, the regularization technique 305 includes creating misspellings for the input word 302 by randomly deleting letters from the input word individually with pre-specified probabilities. Applying the above example, the input word 302, “interspeech,” could become “intrspeech.” Clearly, and with reference to the subgraph 500 of FIG. 5 , any character in the input word could potentially be skipped / deleted to create a misspelling, so all of the potential subword units still exist, regardless of the various probability allocations. Furthermore, by creating misspellings by randomly deleting letters, the routine 300 can learn further relationships between words beyond prefix structure by learning corresponding suffix structures (and general spelling similarities). For example, if the true label is "ing" and the letter "i" is skipped, the model will assign a probability to the subword "ng," thus learning the relationship between "ing" and "ng." Furthermore, because multiple characters can be skipped, the true label at any position can be further into the future, allowing routine 300 to learn good long-term dependencies in the data.

[0049] In a further example, a regularization technique 305 applied prior to executing the subword segmentation routine 300 (e.g., lightweight WPM) includes creating misspellings of the input word 302 by swapping the order of adjacent letter pairs and, for each adjacent letter pair in the input word, pre-specifying a probability to swap the order of letters from adjacent letter pairs in the input word based on a pre-specified probability. Here, any given letter order in the input word is limited to at most one swap. For example, if the original word is "the" and the "t" and "h" are swapped to form "hte," no further swaps are made to the "te" pair after the first swap. As a second example, the input word 302 of "center" could also become "centre."

[0050] 6 shows an exemplary arrangement of operations for a method 600 of subword unit segmentation for improving speech recognition. In operation 602, the method 600 includes receiving an input word 302 to be segmented into a plurality of subword units 119. Operations 604 and 606 are performed while executing the subword segmentation routine 300 for segmenting the input word 302 into a plurality of subword units 119. In operation 604, the method 600 includes accessing a trained vocabulary set 350 of subword units. In operation 606, the method 600 includes selecting a plurality of subword units 119 from the input word 302 by greedily finding the longest subword unit from the input word 302 that is present in the trained vocabulary set 350, until the end of the input word 302 is reached.

[0051] 7 is a schematic diagram of an exemplary computing device 700 that may be used to implement the systems and methods described herein. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality, are meant to be exemplary only and are not meant to limit the embodiments of the invention described and / or claimed herein.

[0052] Computing device 700 includes a processor 710, memory 720, a storage device 730, a high-speed interface / controller 740 connected to memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and storage device 730. Each of components 710, 720, 730, 740, 750, and 670 are interconnected using various buses and may be mounted on a common motherboard or in any other suitable manner. Processor 710 (e.g., data processing hardware 710 of remote server 60) may process instructions executed within computing device 700, including instructions stored in memory 720 or in storage device 730, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to high-speed interface 740. In other embodiments, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and memory types. Additionally, multiple computing devices 700 may be connected together, with each device providing a portion of the required operations (eg, as a server bank, a group of blade servers, or as a multi-processor system).

[0053] Memory 720 (e.g., memory hardware 720 of remote server 60) non-transiently stores information within computing device 700. Memory 720 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), phase-change memory (PCM), as well as disk or tape.

[0054] The storage device 730 can provide mass storage for the computing device 700. In some embodiments, the storage device 730 is a computer-readable medium. In various different embodiments, the storage device 730 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory, or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. In further embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 720, the storage device 730, or memory on the processor 710.

[0055] The high-speed controller 740 manages bandwidth-intensive operations for the computing device 700, while the low-speed controller 760 manages less bandwidth-intensive operations. This duty allocation is merely exemplary. In some embodiments, the high-speed controller 740 is coupled to memory 720, a display 780 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 750 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 760 is coupled to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or to a networking device such as a switch or router, for example, via a network adapter.

[0056] The computing device 700 can be implemented in several different forms as shown in the figure. For example, it can be implemented as a standard server 700a, or it can be implemented multiple times in a cluster of servers 700a, such as as a laptop computer 700b, or as part of a rack server system 700c.

[0057] Various embodiments of the systems and techniques described herein may be realized in digital, electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive and transmit data and instructions from and to a storage system, at least one input device, and at least one output device.

[0058] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0059] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0060] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Processors suitable for executing computer programs include, by way of example, general-purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from, transfer data to, or both, one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal or removable hard disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0061] To provide for user interaction, one or more aspects of the present disclosure may be implemented in a computer having a display device, such as, for example, a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, for displaying information to the user, and optionally, a keyboard and pointing device, such as, for example, a mouse or trackball, by which the user can provide input to the computer. Other types of devices may be used to provide for user interaction as well; for example, feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from devices used by the user, for example, by sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0062] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims. [Explanation of symbols]

[0063] 10 User Devices 12 Data Processing Hardware 14 Memory Hardware 16 Audio System 16a Audio Capture Device 16b Audio Output Device 18 Digital Assistant Interface 19 responses 40 Network 50 Digital Assistant Applications 60 Remote Computing Devices 100 systems 104 users 106 utterances 107 User Interface Generator 108 Audio Subsystem 110 Input audio data, acoustic frames 110T Audio Data 118 ASR System 119 Subword Units 119a~n Subword Unit 120 transcription, speech recognition results 120a First speech recognition result 120b Final speech recognition result 200 Sequence ASR Model 200a Recurrent Neural Network Transducer (RNN-T) Model 200b Listen, Attend, and Spell (LAS) Model 201 Speech recognition results 210 Encoder Network 211 Listener Encoder Module 220 Prediction Network 221 Attendant Module 225 Expression 230 Joint Network 231 RNN-T Decoder 240 softmax layers 300 Subword Segmentation Routines 302 input words 305 Regularization techniques 350 Vocabulary Set 360 Data Storage Device 380 Loss Function 390 Supervised loss term 400 Algorithm 500 subgraphs 600 ways 700 computing devices 700a standard server 700b laptop computer 700c Rack Server System 710 Processor, Data Processing Hardware 720 memory 730 Storage Devices 740 High-Speed ​​Interface / Controller 750 High-Speed ​​Expansion Port 760 Low-Speed ​​Interface / Controller 770 Slow Bus 780 Display 790 Low-Speed ​​Expansion Port

Claims

1. When executed on data processing hardware (710), the data processing hardware (710) receiving an input word (302) to be segmented into a plurality of sub-word units (119); a sub-step of accessing a trained vocabulary set (350) of sub-word units (119); and selecting the plurality of sub-word units (119) from the input word (302) by greedily finding the longest sub-word unit from the input word (302) that is present in the trained vocabulary set (350) until the end of the input word (302) is reached; executing a subword segmentation routine (300) to segment the input word (302) into the plurality of subword units (119) according to A computer-implemented method (600) for performing operations including: The sub-step of selecting the plurality of sub-word units (119) comprises, for each corresponding position of a plurality of different positions of the input word (302): identifying all possible candidate subword units (119) from the input word (302) at the corresponding positions present in the trained vocabulary set (350); and randomly sampling from all of the possible candidate sub-word units (119) by assigning 1-p probability to the longest of the possible candidate sub-word units (119) and dividing the remaining p probability equally among all of the possible candidate sub-word units (119) from the input word (302) at the corresponding positions; A computer-implemented method (600) comprising:

2. 10. The method of claim 1, wherein the operations further comprise, prior to the step of executing the subword segmentation routine, creating typos in the input word by randomly deleting characters from the input word, individually, with a pre-specified probability.

3. The operation includes, prior to the step of executing the subword segmentation routine (300): a substep of pre-specifying probabilities for swapping the order of adjacent letter pairs; for each adjacent character pair in the input word (302), swapping the order of characters from the adjacent character pair in the input word (302) based on the pre-specified probability; 3. The method (600) of claim 1 or 2, further comprising the step of generating a misspelling for the input word (302) by:

4. 4. The method (600) of claim 3, wherein the ordering of any given character in the input word (302) is limited to at most one permutation.

5. The operation is receiving training examples including training audio data (110T) characterizing utterances (106) of the input words (302); processing the training audio data to generate speech recognition results for the utterance of the input words, the speech recognition results including a sequence of hypothesized sub-word units, each hypothesized sub-word unit in the sequence of hypothesized sub-word units being output from the speech recognition model in a corresponding output step; determining a supervised loss term (390) based on the set of hypothesized subword units and the plurality of subword units (119) selected from the input word (302) by the subword segmentation routine (300); updating parameters of the speech recognition model (200) based on the supervised loss term (390); 5. The method (600) of any one of claims 1 to 4, further comprising:

6. 6. The method of claim 5, wherein the speech recognition model includes a recurrent neural network transformer (RNN-T) model architecture, the RNN-T model architecture comprising an audio encoder, a prediction network, and a joint network.

7. 7. The method of claim 6, wherein the audio encoder includes multiple long short-term memory (LSTM) layers.

8. The method of claim 6, wherein the audio encoder includes a plurality of transformer or conformer layers.

9. 6. The method of claim 5, wherein the speech recognition model comprises a listen, attend, spell (LAS) model architecture, including an audio encoder, an attention mechanism, and a decoder.

10. data processing hardware (710); memory hardware (720) in communication with the data processing hardware (710), the memory hardware (720) when executed on the data processing hardware (710) causing the data processing hardware (710) to: receiving an input word (302) to be segmented into a plurality of sub-word units (119); a sub-step of accessing a trained vocabulary set (350) of sub-word units (119); and selecting the plurality of sub-word units (119) from the input word (302) by greedily finding the longest sub-word unit from the input word (302) that is present in the trained vocabulary set (350) until the end of the input word (302) is reached; executing a subword segmentation routine (300) for segmenting said input word (302) into said plurality of subword units (119) according to memory hardware (720) for storing instructions for performing operations including A system (100) comprising: The sub-step of selecting the plurality of sub-word units (119) comprises, for each corresponding position of a plurality of different positions of the input word (302): identifying all possible candidate subword units (119) from the input word (302) at the corresponding positions present in the trained vocabulary set (350); and randomly sampling from all of the possible candidate sub-word units (119) by assigning 1-p probability to the longest of the possible candidate sub-word units (119) and dividing the remaining p probabilities equally among all of the possible candidate sub-word units (119) from the input word (302) at the corresponding positions; A system (100) including:

11. 11. The system (100) of claim 10, wherein the operations further include, prior to the step of executing the subword segmentation routine (300), creating typos in the input word (302) by randomly deleting characters from the input word (302), individually, with a pre-specified probability.

12. The operation includes, prior to the step of executing the subword segmentation routine (300): a substep of pre-specifying probabilities for swapping the order of adjacent letter pairs; for each adjacent character pair in the input word (302), swapping the order of characters from the adjacent character pair in the input word (302) based on the pre-specified probability; The method further includes the step of generating a misspelling for the input word (302) by 12. A system (100) according to claim 10 or 11.

13. 13. The system (100) of claim 12, wherein the ordering of any given character in the input word (302) is limited to at most one permutation.

14. The operation is receiving training examples including training audio data (110T) characterizing utterances (106) of the input words (302); processing the training audio data to generate speech recognition results for the utterance of the input words, the speech recognition results including a sequence of hypothesized sub-word units, each hypothesized sub-word unit in the sequence of hypothesized sub-word units being output from the speech recognition model in a corresponding output step; determining a supervised loss term (390) based on the set of hypothesized subword units and the plurality of subword units (119) selected from the input word (302) by the subword segmentation routine (300); updating parameters of the speech recognition model (200) based on the supervised loss term (390); 14. The system (100) of any one of claims 10 to 13, further comprising:

15. 15. The system of claim 14, wherein the speech recognition model includes a recurrent neural network transformer (RNN-T) model architecture, the RNN-T model architecture comprising an audio encoder, a prediction network, and a joint network.

16. 16. The system of claim 15, wherein the audio encoder includes multiple long short-term memory (LSTM) layers.

17. 16. The system of claim 15, wherein the audio encoder includes multiple transformer or conformer layers.

18. 15. The system of claim 14, wherein the speech recognition model comprises a listen, attend, spell (LAS) model architecture including an audio encoder, an attention mechanism, and a decoder.

Images