Mixing Heterogeneous Loss Types to Improve Keyword Spotting Accuracy

Abstract

A method (800) for training a stored neural network (300) includes receiving a training input audio sequence (400) including a sequence of input frames defining a hot word that initiates a wake-up process on a user device (102). The method further includes obtaining a first label (420, 420a) and a second label (420, 420b) for the training input audio sequence. The method includes using the stored neural network and the training input audio sequence to generate an output (350) indicative of a likelihood that the training input audio sequence includes the hot word. The method further includes determining a first loss (710, 710a) based on the first label and the output. The method includes determining a second loss (710, 710b) based on the second label and the output. The method further includes optimizing the stored neural network based on the first loss and the second loss associated with the training input audio sequence.

Description

[Technical field]

[0001] The present disclosure relates to a system for keyword spotting in streaming audio. [Background technology]

[0002] A voice-enabled environment (e.g., at home, at work, at school, in an automobile, etc.) allows a user to speak queries or commands aloud to a computer-based system that processes and responds to the queries and / or performs functions based on the commands. A voice-enabled environment may be implemented with a network of connected microphone devices distributed throughout various rooms or areas of the environment. These devices may use so-called "hot words" to help identify when a given utterance is directed to the system as opposed to an utterance directed to another individual present in the environment. Thus, devices may operate in a sleep or hibernation state and wake up only if the detected utterance contains the hot word. For a voice-enabled environment to operate optimally, devices in the environment must be able to detect hot words accurately and efficiently. Neural networks have recently emerged as an attractive solution for training models to detect user-spoken hot words in streaming audio. Summary of the Invention

[0003] An aspect of the disclosure provides a method for training a hot word detector using two labels and two loss functions for training data. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations including receiving a training input audio sequence including a sequence of input frames. The sequence of input frames defines a hot word that initiates a wake-up process on a user device. The operations further include obtaining a first label for the training input audio sequence and a second label for the training input audio sequence, the second label being different from the first label. The operations also include using the stored neural network and the training input audio sequence to generate an output indicative of a likelihood that the training input audio sequence includes the hot word. The operations include determining a first loss based on the first label and the output and a second loss based on the second label and the output. The operations further include optimizing the stored neural network based on the first loss and the second loss associated with the training input audio sequence.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the stored neural network includes an encoder and a decoder, each of the encoder and decoder of the stored neural network including sequentially stacked singular value decomposition filter (SVDF) layers. In other implementations, the output is based on a probability distribution of the logits based on the training input audio sequence. In these implementations, the operations further include smoothing the logits before determining the first loss.

[0005] In some additional implementations, determining the first loss includes generating a plurality of encoder windows, where each encoder window of the plurality of encoder windows is associated with one or more phonemes of the hot word. These implementations further include determining the first loss for each encoder window of the plurality of encoder windows. In these implementations, a collection size of the plurality of encoder windows may correspond to an average acoustic length of the hot word.

[0006] Further, optimizing the stored neural network may include generating a weighted average of the first loss and the second loss. In some implementations, the second label is derived from one or more phoneme sequences of the hot word. Alternatively, the first label may be based on a position of the last phoneme of the hot word. Further, the sequence of input frames may each include one or more respective audio features that characterize a phonetic component of the hot word.

[0007] In some exemplary implementations, the first label is a max pooling loss label and the second label is a cross entropy label. In these exemplary implementations, the first loss is a max pooling loss and the second loss is a cross entropy loss.

[0008] Another aspect of the disclosure provides a system for training a hot word detector using two labels and two loss functions for training data. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving a training input audio sequence including a sequence of input frames. The sequence of input frames defines a hot word that initiates a wake-up process on the user device. The operations further include obtaining a first label for the training input audio sequence and a second label for the training input audio sequence, the second label being different from the first label. The operations also include using the stored neural network and the training input audio sequence to generate an output indicative of a likelihood that the training input audio sequence includes the hot word. The operations include determining a first loss based on the first label and the output and a second loss based on the second label and the output. The operations further include optimizing the stored neural network based on the first loss and the second loss associated with the training input audio sequence.

[0009] This aspect may include one or more of the following optional features: In some implementations, the stored neural network includes an encoder and a decoder, each of the encoder and decoder of the stored neural network includes sequentially stacked singular value decomposition filter (SVDF) layers. In other implementations, the output is based on a probability distribution of logits based on the training input audio sequence. In these implementations, the operations further include smoothing the logits before determining the first loss.

[0010] In some additional implementations, determining the first loss includes generating a plurality of encoder windows, where each encoder window of the plurality of encoder windows is associated with one or more phonemes of the hot word. These implementations further include determining the first loss for each encoder window of the plurality of encoder windows. In these implementations, a collection size of the plurality of encoder windows may correspond to an average acoustic length of the hot word.

[0011] Further, optimizing the stored neural network may include generating a weighted average of the first loss and the second loss. In some implementations, the second label is derived from one or more phoneme sequences of the hot word. Alternatively, the first label may be based on a position of the last phoneme of the hot word. Further, the sequence of input frames may each include one or more respective audio features that characterize a phonetic component of the hot word.

[0012] In some exemplary implementations, the first label is a max pooling loss label and the second label is a cross entropy label. In these exemplary implementations, the first loss is a max pooling loss and the second loss is a cross entropy loss.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of an example system for training a memorized neural network and for detecting hot words in a spoken utterance using the trained memorized neural network. [Diagram 2]FIG. 1 is a schematic diagram of components of a typical neural network acoustic encoder used by a model to detect hot words. [Figure 3A] FIG. 2 is a schematic diagram of exemplary components of a stored neural network of the system of FIG. 1. [Figure 3B] FIG. 2 is a schematic diagram of example components of a stored neural network having multiple layers. [Figure 4A] FIG. 1 is a schematic diagram illustrating audio feature and label pairs generated from streaming audio for training a neural network. [Figure 4B] FIG. 1 is a schematic diagram illustrating audio feature and label pairs generated from streaming audio for training a neural network. [Figure 5A] FIG. 2 is a schematic diagram of layers of a stored neural network of the system of FIG. 1. [Figure 5B] FIG. 2 is a schematic diagram of layers of a stored neural network of the system of FIG. 1. [Figure 5C] FIG. 2 is a schematic diagram of an exemplary training process for a stored neural network of the system of FIG. 1. [Figure 6] 5D is a graphical representation of an example of a window used during the training process of FIG. 5C. [Figure 7] FIG. 2 is a schematic diagram of an example training process for the stored neural network of FIG. 1 using two labels and two loss functions. [Figure 8] 1 is a flowchart of an exemplary arrangement of method operations for training a neural network to detect hot words. [Figure 9] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] A voice-enabled device (e.g., a user device running a voice assistant) allows a user to speak queries or commands aloud, process and respond to the queries, and / or perform functions based on the commands. The use of "hot words" (also called "keywords", "attention words", "wake-up phrases / words", "trigger phrases", or "voice action initiation commands"), where conventions reserve certain terms / phrases to be spoken to alert a voice-enabled device, allows the voice-enabled device to distinguish between utterances directed to the system (i.e., to initiate a wake-up process to process one or more terms following the spoken hot word) and utterances directed to individuals in the environment. Typically, a voice-enabled device operates in a sleep state to conserve battery power and does not process incoming audio data unless the spoken hot word is followed by incoming audio data. For example, while in a sleep state, the voice-enabled device captures incoming audio via a microphone and uses a hot word detector trained to detect the presence of a hot word in the incoming audio. When a hot word is detected in the incoming audio, the voice-enabled device initiates a wake-up process to process the hot word and / or any other terms in the incoming audio following the hot word.

[0017] Hot word detection is akin to searching for a needle in a haystack, since the hot word detector must continuously listen to the streaming audio and trigger precisely and instantly when the presence of a hot word is detected in the streaming audio. In other words, the hot word detector is tasked with ignoring the streaming audio unless the presence of a hot word is detected. Typically, neural networks are used by hot word detectors to address the complexity of detecting the presence of a hot word in a continuous stream of audio.

[0018] A hot word detector typically includes three main components: a signal processing front end, a neural network acoustic encoder, and a hand-designed decoder. The signal processing front end may convert raw audio signals captured by a microphone of a user device into one or more audio features formatted for processing by the neural network acoustic encoder component. For example, the neural network acoustic encoder component may convert these audio features into phonemes, and the hand-designed decoder uses a hand-coded algorithm to splice the phonemes together and provide a probability that an audio sequence contains or does not contain a hot word.

[0019] A common method for training a neural network involves providing labeled training samples to the neural network. The training samples are typically pre-screened data inputs that are labeled based on the desired output of the neural network. For example, in the case of a hot word detector, the training samples are labeled with an indication of the presence of the hot word (e.g., "1" if the hot word is present in the training sample, "0" otherwise). The neural network analyzes the training samples and generates an output or prediction that is then compared to a predefined target output (i.e., label) to determine a loss using a loss function. The loss indicates the accuracy of the output compared to the label. The loss is then fed to the neural network, which adjusts one or more weights, values, or parameters based on the loss.

[0020] When training a hot word detector, the training samples may include audio sequences, and the neural network may output an indication or probability that the audio sequence contains a hot word. Although this training process seems straightforward, there are many different ways to label audio sequences, and each labeling convention results in a unique loss. Similarly, each resulting loss adjusts the hot word detector in a specific way. For example, a simple cross-entropy based loss uses labels derived from phoneme sequences. Cross-entropy based losses are highly sensitive to the positional alignment of all sub-phonemes that comprise a keyword, and thus the resulting trained hot word detector uses a significant portion of its resources to learn to predict the exact position to signal the detection of a pattern. In another example, a max pooling technique uses labels derived only from the position of the last phoneme of a keyword. Thus, unlike the cross-entropy example above, the max pooling technique does not rely on the positional information of other sub-phonemes, and the resulting hot word detector is more stable than a hot word detector trained using a cross-entropy based loss, since it can ignore noise in the positional alignment. However, each of the above examples has advantages and disadvantages when compared to each other and to other techniques.

[0021] The embodiments herein are directed to an end-to-end hot word spotting system (also referred to as a "keyword spotting system") that trains a hot word detector using two distinct target labels for each sample in a training input set. Similarly, for each training sample, the two target labels are each used to determine two distinct losses. The hot word detector then receives a loss based on both of the two losses and adjusts accordingly. By using two losses that are determined differently using two different labels, the hot word detector can be fine-tuned using multiple techniques. For example, by simultaneously training a hot word detector using loss functions based on cross entropy and max pooling, a hot word detector that is more accurate and efficient than a hot word detector trained with either technique individually can be obtained.

[0022] 1, in some implementations, an exemplary system 100 includes one or more user devices 102, each associated with a respective user 10 and communicating with a remote system 110 via a network 104. Each user device 102 may correspond to a computing device such as a mobile phone, a computer, a wearable device, a smart appliance, a smart speaker, etc., and includes data processing hardware 103 and memory hardware 105. The remote system 110 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic computing resources 112 (e.g., data processing hardware) and / or storage resources 114 (e.g., memory hardware). The user device 102 receives a trained stored neural network 300 from the remote system 110 via the network 104 and executes the trained stored neural network 300 to detect hot words in the streaming audio 118. The trained, stored neural network 300 may reside in a hot word detector 106 (also called a hot worder) of the user device 102, which is configured to detect the presence of hot words in the streaming audio 118 without performing semantic analysis or speech recognition processing on the streaming audio 118. Optionally, the trained, stored neural network 300 may additionally or alternatively reside in an automatic speech recognizer (ASR) 108 of the user device 102 and / or the remote system 110 to confirm that the hot word detector 106 has correctly detected the presence of hot words in the streaming audio 118.

[0023] In some implementations, the data processing hardware 112 trains the stored neural network 300 using training samples 400 obtained from the annotated utterance pool 130. The training samples may include a first label 420, 420a and a second label 420, 420b. That is, each training sample may be annotated with two distinct labels 420a, 420b. The annotated utterance pool 130 may reside in the memory hardware 114 and / or in some other remote memory location(s). In the illustrated example, when the user 10 speaks an utterance 120 that includes a hot word (e.g., "Hey Google") that is captured as streaming audio 118 by the user device 102, a stored neural network 300 running on the user device 102 is configured to detect the presence of the hot word in the utterance 120 so as to initiate a wake-up process on the user device 102 to process the hot word and / or one or more other terms (e.g., queries or commands) that follow the hot word in the utterance 120. In additional implementations, the user device 102 transmits the utterance 120 to the remote system 110 for further processing or validation (e.g., with another, potentially more computationally intensive stored neural network 300).

[0024] In the illustrated example, the stored neural network 300 includes an encoder portion 310 and a decoder portion 311, each including a layered topology of singular value decomposition filter (SVDF) layers 302. The SVDF layers 302 provide memory to the neural network 300 by providing memory capacity for each SVDF layer 302 such that the memory capacity of all SVDF layers 302 additionally constitutes an overall fixed memory for the neural network 300 to store only the fixed length of time of streaming audio 118 required to capture audio features 410 (FIGS. 4A and 4B) characterizing hot words. It should be understood that this stored neural network 300 architecture is exemplary and that any stored neural network 300 architecture may be substituted.

[0025] In some implementations, the stored neural network 300 is trained using multiple labels 420, 420a-b to generate a respective loss 710, 710a-b for each corresponding label 420a-b. The process of training the neural network 300 with multiple labels 420 is described in more detail below (FIG. 7).

[0026] Referring now to FIG. 2, a typical hot word detector uses a memoryless neural network acoustic encoder 200. Because the network 200 has no memory, each neuron 212 of the acoustic encoder 200 must simultaneously accept as input every audio feature of every frame 210, 210a-d, of the spoken utterance 120. Note that each frame 210 can have any number of audio features, each of which is accepted as input by a neuron 212. Such an arrangement requires a substantial size of the neural network acoustic encoder 200, which increases dramatically as the fixed long time period increases and / or as the number of audio features increases. The output of the acoustic encoder 200 results in, for example, the probability of each phoneme of a detected hot word. The acoustic encoder 200 must then rely on a hand-coded decoder to process the output of the acoustic encoder 200 (e.g., splice together the phonemes) to generate a score (i.e., an estimate) indicative of the presence of a hot word.

[0027] 3A and 3B, in some implementations, a singular value decomposition filter (SVDF) neural network 300 (also referred to as a stored neural network) has any number of neurons / nodes 312, with each neuron 312 accepting only a single frame 210, 210a-d, of the spoken utterance 120 at a time. That is, where each frame 210 constitutes, for example, 30 ms of audio data, each frame 210 is input to the neuron 312 approximately every 30 ms (i.e., time 1, time 2, time 3, time 4, etc.). FIG. 3A illustrates each neuron 312 including a two-stage filtering mechanism, a first stage 320 (i.e., stage 1 feature filter) that performs filtering on the feature dimension of the input, and a second stage 340 (i.e., stage 2 time filter) that performs filtering on the time dimension of the output of the first stage 320. Thus, the stage 1 feature filter 320 performs feature filtering on only the current frame 210. The results of the processing are then placed in the memory component 330. In these examples, the size of the memory component 330 is configurable on a per node level or per layer level. After the stage 1 feature filter 320 processes a given frame 210 (e.g., by filtering the audio features in the frame), the filtered results are placed in the next available memory location 332, 332a-d in the memory component 330. When all memory locations 332 are full, the stage 1 feature filter 320 overwrites the memory location 332 that stores the oldest filtered data in the memory component 330. For illustrative purposes, FIG. 3A shows a memory component 330 of size 4 (four memory locations 332a-d) and four frames 210a-d, however, it should be noted that due to the nature of hot word detection, the system 100 typically continuously monitors the streaming audio 118 such that each neuron 312 "slides" along the pipeline or processes the frames 210 as well as the pipeline.In other words, if each stage includes N feature filters 320 and N temporal filters 340 (each corresponding to the size of the input feature frame 210), the layer is akin to computing N×T (T equals the number of frames 210 in a fixed period) convolutions of the feature filters by sliding each of the N filters 320, 340 over the input feature frame 210 with a stride of the size of the feature frame. For example, since the example shows the memory component 330 in capacity after the stage 1 feature filter outputs filtered audio features associated with frame 4 (F4) 210d (for time 4), the stage 1 feature filter 320 places filtered audio features associated with the subsequent frame 5 (F5) (for time 5) in the memory 330 by overwriting the filtered audio features associated with frame 1 (F1) 210a in memory location 332a. In this manner, the stage 2 temporal filter 340 applies filtering to the previous T-1 (T again equal to the number of frames 210 in the fixed period) filtered audio features output from the stage 1 feature filter 320.

[0028] The stage 2 temporal filter 340 then filters each filtered audio feature stored in the memory 330. For example, FIG. 3A shows that each time the stage 1 feature filter 320 stores a new filtered audio feature in the memory 330, the stage 2 temporal filter 340 filters the audio feature in each of the four memory locations 332. In this way, the stage 2 temporal filter 340 is always filtering a number of past frames 210, the number of which is proportional to the size of the memory 330. Each neuron 312 is part of a single SVDF layer 302, and the neural network 300 may include any number of layers 302. The output of each stage 2 temporal filter 340 is passed to the input of a neuron 312 in the next layer 302. The number of layers 302 and the number of neurons 312 per layer 302 are fully configurable and depend on the available resources and the desired size, power, and accuracy. The present disclosure is not limited to the number of SVDF layers 302 or the number of neurons 312 in each SVDF layer 302.

[0029] 3B, each SVDF layer 302, 302a-n (or simply "layer") of the neural network 300, in some implementations, is connected such that the output of the previous layer is accepted as an input to the corresponding layer 302. In some examples, the final layer 302n outputs a probability score 350 that indicates the probability that the utterance 120 contains a hot word.

[0030] In the illustrated example SVDF network 300, the layer design stems from the idea that a densely connected layer 302 processing a sequence of input frames 210 can be approximated by using the singular value decomposition of each of its nodes 312. The approximation is configurable. For example, a rank-R approximation means extending a new dimension R to the layer's filters, where stage 1 occurs independently and stage 2 adds the outputs of all ranks together before passing through the nonlinearity. In other words, the SVDF decomposition of the nodes 312 of a densely connected layer of matching dimension may be used to initialize the SVDF layer 302, which provides a principled initialization and improves the generalization quality of the layer. In essence, the "power" of the larger densely connected layer is transferred to the potentially much smaller SVDF (depending on the rank). However, it should be noted that the SVDF layer 302 does not require initialization to outperform densely connected layers or even convolutional layers with the same or even more operations.

[0031] In some implementations, the system 100 includes a stateful, stackable neural network 300, where each neuron 312 of each SVDF layer 302 includes a first stage 320 associated with filtering audio features and a second stage 340 associated with filtering an output of the first stage 320 with respect to time. Specifically, the first stage 320 is configured to perform filtering on one or more audio features on one audio feature input frame 210 at a time and output the filtered audio features to a respective memory component 330, where the stage 1 feature filter 320 receives one or more audio features associated with the time frame 210 as input for processing and outputs the processed audio features to the respective memory component 330 of the SVDF layer 302. The second stage 340 is then configured to perform filtering on all of the filtered audio features output from the first stage 320 and present in the respective memory component 330. For example, if each memory component 330 is equal to 8, the second stage 340 will pull up to 8 last filtered audio features present in the memory component 330 output from the first stage 320 during individual filtering of audio features in a sequence of 8 input frames 210. When the first stage 320 fills the corresponding memory component 330 to capacity, the memory location 332 containing the oldest filtered audio feature is overwritten (i.e., first in, first out). Thus, depending on the capacity of the memory component 330 in the SVDF neuron 312 or layer 302, the second stage 340 may store several past outputs processed by the first stage 320 of the corresponding SVDF layer 302. Furthermore, because the memory components 330 in the SVDF layer 302 are additive, the memory components 330 in each SVDF neuron 312 and layer 302 also include the memory of the respective preceding SVDF neuron 312 and layer 302, and thus the overall receptive field of the memorized neural network 300 is expanded.For example, in a topology of a neural network 300 having four SVDF layers 302, each having a single neuron 312 with a memory component 330 equal to 8, the last SVDF layer 302 contains a sequence of up to 32 final audio feature input frames 210 that have been individually filtered by the neural network 300. However, it should be noted that the amount of memory can be configured per layer 302 or even per node 312. For example, the first layer 302a may be assigned 32 positions 332, and the last layer 302 may be configured with 8 positions 332. As a result, the stacked SVDF layers 302 allow the neural network 300 to process audio features of only one input time frame 210 (e.g., 30 milliseconds of audio data) at a time, and incorporate some filtered audio features in the past that capture a fixed long time period required to capture a specified hot word in the streaming audio 118. In contrast, a memoryless neural network 200 (as shown in FIG. 2) requires its neurons 212 to process all of the audio feature frames covering a fixed long time (e.g., 2 seconds of audio data) at once to determine the probability that the streaming audio contains the presence of a hot word. This dramatically increases the overall size of the network. Furthermore, while a recurrent neural network (RNN) using a long short-term memory (LSTM) provides memory, the RNN-LSTM forces the neuron, which has an essentially infinite memory, to continuously update the neuron's state after each processing instance, thereby hindering the ability to remember a finite number of past processed outputs, where each new output rewrites the previous output (when the fixed-size memory reaches capacity). In other words, the SVDF network does not regress the output to the state (memory) and does not rewrite all the state at every iteration. Instead, the memory maintains the state of each estimation run isolated from subsequent runs, rather than pushing and popping new entries based on the memory size configured for the layer.

[0032] 4A and 4B, in some implementations, the stored neural network 300 is trained on a plurality of training input audio sequences 400 (i.e., training samples), each of which includes a sequence of input frames 210, 210a-n and two or more labels 420a-b assigned to the input frames 210. Each input frame 210 includes one or more respective audio features 410 that characterize a hot word's phonetic component 430, and each label 420 indicates a probability that the one or more audio features 410 of the respective input frame 210 include the hot word's phonetic component 430. In some examples, the audio features 410 of each input frame 210 are converted from the raw audio signal 402 of the audio stream 118 during a pre-processing stage 404. The audio features 410 may include one or more log filter banks. Thus, the pre-processing stage may segment the audio stream 118 (or spoken utterance 120) into a sequence of input frames 210 (e.g., 30 ms each) and generate a separate log filter bank for each frame 210. For example, each frame 210 may be represented by 40 log filter banks. Furthermore, each successive SVDF layer 302 receives as input the time-filtered audio features 410 output from the previous SVDF layer 302.

[0033] In the illustrated example, each training input audio sequence 400 is associated with training samples that include annotated (i.e., with labels 420a-b) utterances that include a designated hot word that occurs within a fixed long period of time (e.g., 2 seconds). The stored neural network 300 may also be optionally trained against annotated utterances 400 that do not include the designated hot word, or that include the designated hot word but span a time period longer than the fixed long period of time, and thus are not falsely detected because the fixed memory does not store data outside the fixed long period of time. In some examples, the fixed long period of time corresponds to the time it takes a typical speaker to speak the designated hot word to invoke the user device 102 to process the spoken query and / or voice command. For example, if the designated hot word includes the phrases "Hey Google" or "Ok Google," a fixed long period of time set equal to 2 seconds is likely sufficient, since even slow speakers typically do not take more than 2 seconds to speak the designated phrase. Thus, since it is only important to detect occurrences of the specified hot words in the streaming audio 118 for a fixed long time, the neural network 300 includes a fixed memory in an amount proportional to the amount of audio that spans the fixed time (e.g., 2 seconds). Thus, the fixed memory of the neural network 300 allows the neurons 312 of the neural network to filter audio features 410 (e.g., log filter bank) from one input frame 210 (e.g., a 30 ms time window) of the streaming audio 118 at a time, while storing the most recent filtered audio features 410 that span the fixed long time, and remove or delete any filtered audio features 410 outside the fixed long time from the current filtering iteration.Thus, if the neural network 300 has a memory depth of, for example, 32, the first 32 frames processed by the neural network 300 will fill the memory component 330 to capacity, and with each new output after the first 32, the neural network 300 will remove the oldest processed audio feature from the corresponding memory location 332 of the memory component 330.

[0034] 4A, for end-to-end training, the training input audio sequence 400a includes a label 420a that may be applied to each input frame 210. In some examples, if the training sample 400a includes a hot word, the target label 420a associated with the target score (e.g., "1") is applied to one or more input frames 210 that include audio features 410 that characterize speech components 430 at or near the end of the hot word. For example, if the speech components 430 of the hot word "OK Google" are divided into "ou", "k", "eI", "<silence>", "g", "u", "g", "@", and "l", the target label of the number "1" is applied to all input frames 210 that correspond to the letter "l" (i.e., the last component 430 of the hot word), which are part of the required sequence of speech components 430 of the hot word. In this scenario, all other input frames 210 (not associated with the last speech component 430) are assigned a different label (e.g., "0"). Thus, each input frame 210 includes a corresponding input feature and label pair 410, 420a. The input features 410 are typically one-dimensional tensors, corresponding for example to a mel or log filter bank, computed from the input audio over the input frame 210.

[0035] The exemplary labels 420a focus on the position of the last phoneme of the hot word and do not rely on the position information of other sub-phonemes (hence the label "0" for phonetic components that are not "l"). Typically, this type of label 420a is associated with a max pooling loss, which does not rely on the exact position of the target pattern but tries to define the presence of a pattern within a defined interval. The labels 420a are generated from the annotated utterances 400a, where each input feature tensor 410 is assigned a phonetic class by a forced alignment step (i.e., the label "1" is given to the pair corresponding to the last class belonging to the hot word and "0" is given to all the rest). Thus, the training input audio sequence 400a includes binary labels assigned to a sequence of input frames. The annotated utterances 400a, or training input audio sequence 400a, correspond to the training samples 400 obtained from the annotated utterance pool 130 of FIG. 1.

[0036] In another example, Figure 4B includes a training input audio sequence 400b including labels 420b associated with scores that increase along the sequence of input frames 210 as the number of audio features 410 that characterize (match) the hot word's phonetic components 430 progresses. For example, if the hot word includes "Ok Google", the input frame 210 including the respective audio features 410 that characterize the first phonetic components "o" and "k" is assigned the label 420b of "1", the input frame 210 including the respective audio features 410 that characterize the last phonetic component "l" is assigned the label 420b of "5". The input frames 210 including the respective audio features 410 that characterize the intermediate phonetic components 430 are assigned the labels 420b of "2", "3", and "4".

[0037] In additional implementations, the number of positive labels 420b is increased. For example, starting from the first frame 210 that includes the audio feature 410 that characterizes the last speech component 430 of the hot word, a fixed amount of "1" labels 420b are generated. In this implementation, when the configured number of positive labels 420b (e.g., "1") is large, positive labels 420b may be applied to frames 210 where a non-positive label 420b (e.g., "0") would have been applied otherwise. In other examples, the starting position of the positive labels 420b is modified. For example, the labels 420b may be shifted to start at either the beginning, middle, or end of the segment of the frame 210 that includes the last keyword speech component 430. In yet other examples, weighted losses are associated with the input sequence. For example, weighted loss data is added to the input sequence that allows the training procedure to reduce losses (i.e., error gradients) caused by small misalignments. Specifically, in a frame-based loss function, losses may arise from either misclassification or misalignment. To reduce the loss, the neural network 300 predicts both the correct label 420b and the correct position (timing) of the label 420b. Even if the network 300 detects a keyword at some point, if it is not perfectly aligned with the given target label 420b, the result may be considered erroneous. Thus, weighting the loss is particularly useful for frames 210 where the likelihood of misalignment during the forced alignment stage is high. The exemplary label 420b is typically associated with a cross-entropy loss, resulting in a model that is highly sensitive to positional alignment of all subphonemes of the keyword.

[0038] As a result of training using any of the training input audio sequences 400a, 400b of FIGS. 4A-4B, the neural network 300 is optimized (using the determined loss) to generate an output 350 indicating whether a hot word(s) is present in the streaming audio 118. In some examples, the network 300 is trained in two stages. Now referring to FIG. 5A, a schematic diagram 500a shows an encoder portion (or simply "encoder") 310a of the neural network 300, which includes eight layers trained individually to produce, for example, acoustic posterior probabilities. In addition to the SVDF layer, the network 300 may include, for example, bottleneck, softmax, and / or other layers. To train the encoder 310a, label generation assigns separate classes to all phonetic components of the hot words (plus an "epsilon" target for silence and everything that is not a hot word). The decoder portion (or simply "decoder") 311a of the neural network 300 is then trained by creating a topology whose first portion (i.e., layers and connections) matches that of the encoder 310a, and a selected checkpoint from that encoder 310a of the neural network 300 is used to initialize the neural network 300. The training is specified to "freeze" (i.e., not update) the parameters of the encoder 310a, thus adjusting only the decoder 311a portion of the topology. This naturally produces a single spotter neural network, even though it is the product of two staggered training pipelines. Training using this method is particularly useful for models that tend to exhibit overfitting to parts of the training set.

[0039] Alternatively, the neural network 300 is trained end-to-end from scratch. For example, the neural network 300 accepts features directly (as in the training of the encoder 310a described above), but instead uses the output of a binary target label 420a (i.e., "0" or "1") for use in training the decoder 311a. Such an end-to-end neural network 300 may use any topology. For example, as shown in FIG. 5B, a schematic diagram 500b shows a topology of the neural network 300 for the encoder 310b and the decoder 311b similar to the topology of FIG. 5A, except that the encoder 310b does not include an intermediate softmax layer. As with the topology of FIG. 5A, the topology of FIG. 5B may use a pre-trained encoder checkpoint with a fitness factor to adjust how the decoder 311b portion is adjusted (e.g., if the fitness factor is set to 0, it is equivalent to the topology of FIG. 5A). This end-to-end pipeline tends to outperform the separately trained encoder 310a and decoder 311a of FIG. 5A when the entire parameters of the topology are tuned, especially for smaller sized models that are less prone to overfitting.

[0040] Thus, the neural network 300 may avoid the use of manually tuned decoders. Manually tuning a decoder increases the difficulty of modifying or adding hot words. A single stored neural network 300 may be trained to detect multiple different hot words, as well as the same hot word across two or more locations. Furthermore, detection quality is degraded compared to networks specifically optimized for hot word detection trained with potentially millions of examples. Furthermore, a typical manually tuned decoder is more complex than a single neural network that performs both encoding and decoding. Conventional systems tend to be over-parameterized, consume significantly more memory and computation than comparable end-to-end models, and do not fully utilize neural network acceleration hardware. Furthermore, manually tuned decoders have problems with accented speech, making it extremely difficult to create a detector that can work across multiple regions and / or languages.

[0041] The memorized neural network 300 outperforms a simple fully-connected layer of the same size, but also benefits from optional initialization of parameters from a pre-trained fully-connected layer. The network 300 allows fine-grained control over the amount of recall from the past. This results in the RNN-LSTM outperforming for certain tasks that do not benefit (and in fact are harmed) from paying attention to a theoretically infinite past (e.g., continuously listening to streaming audio). However, the network 300 can work in concert with the RNN-LSTM, typically utilizing an SVDF in lower layers to filter past noisy low-level features and an LSTM in higher layers. The number and computation of parameters is precisely controlled, assuming several relatively small filters include the SVDF. This is helpful when choosing a tradeoff between quality and size / computation. Moreover, because of this quality, the network 300 allows for the creation of very small networks that outperform other topologies such as simple convolutional neural networks (CNNs) that operate at a larger granularity.

[0042] 5C and 6, in some configurations, rather than optimizing the neural network 300 to generate a probability score 350 indicating the likelihood that a hot word(s) is present in the streaming audio 118 using a cross-entropy loss, the neural network 300 is optimized using a smoothed max-pooling loss. Here, similar to the example shown in FIG. 5A and FIG. 5B, this approach involves jointly training the encoders 310, 310c and the decoders 311, 311c. With this smoothed max-pooling loss approach, the neural network 300 can be trained to detect not only a portion of a hot word (e.g., with the encoder 310c) but also the entire hot word (e.g., with the decoder 311c). By using the smoothed max-pooling loss approach, this approach is independent of the frame labels 420a-b and may be suitable for implementations such as on-device learning (e.g., for the user device 102).

[0043] In hot word detection, the exact location of the hot word is generally less important than the actual presence of the hot word. Thus, alignment of the frame labels 420 may cause hot word detection errors (i.e., hot word detection is potentially impaired). This alignment may be particularly problematic when the frame labels 420 have inherent uncertainty caused by noise or accents of certain words. In the case of frame labels 420, the training input audio sequence 400 often contains intervals of repeated similar or identical frame labels 420, called runs. For example, both FIG. 4A and FIG. 4B contain runs of "0". These runs indicate that when training the network 300, the network 300 should make strong learning associations for the generation of the output 350. In contrast, smoothed max pooling approaches (e.g., as shown in FIG. 5C and FIG. 6) avoid using the frame labels 420 to specify the exact activation location (i.e., to specify the timing).

[0044] For the smoothed max pooling loss approach, in some examples, an initial loss is defined for both the encoder 310c and the decoder 311c, and then the initial losses of the encoder 310c and the decoder 311c are optimized simultaneously. Max pooling refers to a sample-based discretization process in which the dimensionality of an input is reduced by applying a maximum filter. In some examples, the training process 500c using the smoothed max pooling technique includes a smoothing operation 510, 510e-d, and a max pooling operation 520, 520e-d. In these examples, the smoothing operation 510 occurs before the max pooling operation 520. Here, during the smoothing operation 510, the training process 500c performs temporal smoothing on the frame 210. For example, the training process 500c smoothes the logits 502, 502e-d corresponding to the frame 210. The logits generally refer to vectors or other raw prediction forms output from one or more SVDF layers 302. The logits 502 serve as inputs to a softmax portion of the encoder 310 and / or decoder 311 such that the encoder 310 and / or decoder 311 generate output probabilities based on the input of one or more logits 502. For example, the logits 502 are a form of unnormalized prediction data, and the softmax normalizes the logits 502 into probabilities (e.g., probabilities of hot words).

[0045] By having the smoothing operation 510 before the max pooling operation 520, the training process 500c trains the network 300 to be more stable against small fluctuations and temporal shifts in the streaming audio 118. This increased stability is in contrast to other training approaches that may use some form of max pooling operation without a temporal smoothing operation. For example, other training approaches may use max pooling in the time domain and determine the cross entropy loss with respect to the logits 502 of the frame 210 with the maximum activation. By introducing the temporal smoothing operation 510 before the max pooling operation 520, the training process 500c of the network 300 may result in smooth activations and stable peak values.

[0046] During the max pooling operation 520, the training process 500c determines a smoothed max pooling loss, where the loss represents the difference between what the output distribution the network 300 thinks theoretically should be and the actual output distribution, where the smoothed max pooling loss may be determined by the following formula: loss = loss + +loss - (1)

[0047]

number

[0048] where X t is the d-dimensional spectral feature, and y i (X t ,W) represents the i-th dimension of the softmax output of the neural network, W is the network weight, and C t is the frame label 420 at frame t (e.g., frame 210), s(t) is a smoothing filter,

[0049]

number

[0050] is the convolution over time,

[0051]

number

[0052] defines the start and end times of the i-th max pooling window interval.

[0053] Continuing to refer to FIG. 5C, both the encoder 310c and the decoder 311c undergo a training process 500c that uses a smoothed max pooling technique. For example, FIG. 5C shows the encoder 310c including smoothing operations 510, 510e and max pooling operations 520, 520e. During the max pooling operation 520e of training 500c, the encoder 310c learns sequences of sound portions (e.g., phonetic components of the audio features 410) that define hot words, where this learning may occur in a semi-supervised manner. In some examples, the max pooling operation 510e during training 500c trains fixed-length hot words (e.g., hot words of a desired length or hot words of an average length) through max pooling windows 310w, 310w. 1~n This occurs by dividing the

[0054] For example, Figure 6 shows n consecutive windows 310w spanning expected hot word positions. A max pooling operation 510e then determines the max pooling loss in each window 310w. In some implementations, the max pooling loss in each window 310w is defined by the following formula:

[0055]

number

[0056] where "e" corresponds to the variable of encoder 310c, and ω end corresponds to the end of the hot word, and offset refers to the time offset of the window 310w.

[0057] In some examples, the number of windows 310w and / or the size 310w of each window 310w may be s are parameters that can be adjusted during the training process 500c. These parameters determine whether the number of windows 310w, “n”, approximates the number of distinguishable sound parts (e.g., phonemes) and / or the size 310w of the windows 310w. smultiplied by the number of windows 310w, “n”, may be adjusted to approximately match the fixed length of the hotword. s In addition to being adjustable, the hot word end point ω end The encoder offset that offsets the sequence of window 310w from Offset e A variable called may also be adjustable during training 500c of the encoder 310c.

[0058] Similar to the encoder 310c, in the training process 500c, the decoder 311c includes smoothing operations 510, 510d and max pooling operations 520, 520d. Generally speaking, the training process 500c trains the decoder 311c to generate strong activations (i.e., high probability of hot word detection) for input frames 210 that contain audio features 410 at or near the end of a hot word. Due to the nature of max pooling loss, the max pooling loss value is calculated based on the time when the decoder window 311w is at or near the actual end point ω of the hot word. end As long as it contains the hot word end point ω end During the max pooling operation 520d of the decoder 311c, the training process 500c calculates the hot word end point ω end Determine the maximum pooling loss for a window 311w that contains

[0059]

number

[0060] Here, offset d And win size d is the expected end point of the hot word, ω end may be adjustable parameters to include

[0061] With continued reference to FIG. 6, the decoder window 311w includes:

[0062]

number

[0063] from

[0064]

number

[0065] The interval is shown as spanning ω end When ω is large enough to contain ω, the smoothed max pooling loss approach allows the network 300 to learn (e.g., in a semi-supervised manner) the optimal locations of the strongest activations. In some examples, the training process 500c may include ω based on the word-level alignment to determine the hot word endpoints ω end In some implementations, the hot word end point ω end is determined based on the output of the encoder 310.

[0066] In contrast to some end-to-end networks 300 with joint training, where the encoder 310 may be trained first and then the decoder 311 may be trained while the model weights of the encoder 310 are frozen, the smoothed max pooling approach jointly trains the encoder 310c and the decoder 311c simultaneously without such freezing. Because the encoder 310c and the decoder 311c are jointly trained during the training process 500c using the smoothed max pooling loss, the relative importance of each loss may be controlled by an adjustable parameter α. For example, the total loss, which refers to the loss at the encoder 310c and the loss at the decoder 311c, has the relationship expressed by the following equation: Total loss = α * loss e +loss d (10)

[0067] 7, a training process 700 for a stored neural network 300 includes using a first label 420a (e.g., a cross-entropy label 420) and a second label 420b (e.g., a max pooling loss label) and a first loss function 705, 705a and a second loss function 705, 705b to generate a first loss 710, 710a and a second loss 710, 710b, respectively. Optionally, the process 700 begins by providing a training input audio sequence 400 to the stored neural network 300, where the data of the training input audio sequence 400 is labeled using both labels 420a-b. For example, a single training input audio sequence 400 is labeled using the first label 420a and the second label 420b as described above with respect to FIGS. 4A and 4B. The example labels 420a, 420b are for illustrative purposes and are not intended to be limiting, as any suitable labeling rule applicable to determining the loss 710 may be used in the training process 700.

[0068] Upon receiving the training input audio sequence 400, the stored neural network 300 may generate an output 350 (i.e., a probability score 350). The stored neural network 300 may process the training input audio sequence 400 in any manner described with respect to any of FIGS. 2-6, or in any other suitable manner for processing audio data to determine the likelihood that a hot word is present in the training input audio sequence 400. In some implementations, the output 350 is used by each of two loss functions 705. That is, a first loss function 705a receives the output 350 and the labels 420a to determine a first loss 710a. Similarly, a second loss function 705b receives the output 350 and the labels 420b to determine a second loss 710b. In particular, the losses 710 are determined from the same output 350 by using two different labels 420a, 420bb of the same training input audio sequence 400 and two different loss functions 705a, 705b, respectively. The loss function 705 may determine the losses 710 in any manner as described with respect to any of Figures 2-6. In some examples, the first loss function 705a is a max pooling loss function and the second loss function 705b is a cross entropy loss function. In other implementations, a single loss function 705 receives the output 350 and the labels 420 and generates a respective loss 710 based on each label 420. The loss function 705 may implement any suitable technique, such as regression loss, mean squared error, mean squared log error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc.

[0069] In some implementations, the losses 710a, 710b are fed directly into the stored neural network 300 during the training process 700. In other implementations, the losses 710a, 710b are combined or weighted together to generate a combined loss 710, 710c, which is processed by the stored neural network 300. In some implementations, the losses are averaged using a weighted average formula. For example, the first loss 710a and the second loss 710b may be defined as follows: First loss = L1[f(X,),Y1] (11) Second loss = L2[f(X,),Y2] (12)

[0070] where X is the output 350, L1 is the first loss function 705a, Y1 is the label 420a, L2 is the second loss function 705b, and Y2 is the label 420b. In these examples, the combined loss 710c is represented by: Coupling loss=alpha*L1[f(X,theta),Y2]+beta*L2[f(X,theta),Y2] (13)

[0071] where alpha and beta are scalar hyperparameters. The first loss 710a and the second loss 710b may be combined in any other manner (e.g., by addition, multiplication, etc.).

[0072] The examples herein show training a neural network 300 using a training input audio sequence 400 annotated with two labels 420a,b. A first loss function 705a uses the output 350 and the label 420a to generate a first loss 710a. A second loss function 705b uses the output 350 and the label 420b to generate a second loss 710b. The neural network is trained, updated, or fine-tuned using both the first loss 710a and the second loss 710b. It should be understood that these examples are non-limiting and that any number of labels 420 and any number of respective loss functions 705 may generate any number of losses to train any suitable neural network 300.

[0073] 8 is a flowchart of an example configuration of operations for a method 800 of training a neural network 300 using multiple labels 420 and multiple loss functions 705. At operation 802, the method 800 includes receiving a training input audio sequence 400 including a sequence of input frames, where the sequence of input frames defines a hot word that initiates a wake-up process on the user device 102. At operation 804, the method 800 includes obtaining a first label 420a (e.g., a max pooling label) and a second label 420b (e.g., a cross entropy label) for the training input audio sequence 400. At operation 806, the method 800 includes using the stored neural network 300 and the training input audio sequence 400 to generate an output 350 indicative of a likelihood that the training input audio sequence 400 includes the hot word. At operation 808, the method 800 includes determining a first loss 710a (e.g., a max pooling loss) based on the first label 420a and the output 350. At operation 810, the method 800 includes determining a second loss 710b (e.g., a cross entropy loss) based on the second label 420b and the output 350. At operation 814, the method 800 includes optimizing the stored neural network 300 based on the first loss 710a and the second loss 710b associated with the training input audio sequence 400.

[0074] As used herein, a software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0075] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0076] 9 is a schematic diagram of an exemplary computing device 900 that may be used to implement the systems and methods described herein. Computing device 900 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are meant to be exemplary only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0077] Computing device 900 includes a processor 910, a memory 920, a storage device 930, a high-speed interface / controller 940 that connects to memory 920 and a high-speed expansion port 950, and a low-speed interface / controller 960 that connects to a low-speed bus 970 and storage device 930. Each component 910, 920, 930, 940, 950, and 960 are interconnected using various buses and may be mounted on a common motherboard or otherwise present as needed. Processor 910 may process instructions for execution within computing device 900, including instructions stored in memory 920 or storage device 930, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 970 connected to high-speed interface 940. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Multiple computing devices 900 may also be connected (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system) with each device providing a portion of the required operations.

[0078] The memory 920 stores information non-transiently within the computing device 900. The memory 920 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 920 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 900. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0079] The storage device 930 can provide mass storage for the computing device 900. In some implementations, the storage device 930 is a computer-readable medium. In various different implementations, the storage device 930 can be an array of devices, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 920, the storage device 920, or a memory on the processor 910.

[0080] The high-speed controller 940 manages bandwidth-intensive operations of the computing device 900, and the low-speed controller 960 manages lower bandwidth-intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 940 is coupled to the memory 920, the display 980 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 950 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 960 is coupled to the storage device 930 and the low-speed expansion port 990. The low-speed expansion port 990 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled, for example, via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0081] The computing device 900 may be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 900a, or multiple times within a group of such servers 900a, as a laptop computer 900b, or as part of a rack server system 900c.

[0082] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from the storage system and to transmit data and instructions to the storage system.

[0083] These computer programs (also known as programs, software, software applications, or codes) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0084] The processes and logic flows described herein may be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data to generate output. The processors and logic flows may also be executed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any type of digital computer. In general, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from them, transfer data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0085] To provide for interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, spoken language, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0086] Although several implementations have been described, it is understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (800) that, when executed by data processing hardware (103, 112), causes the data processing hardware (103, 112) to perform operations, the operations comprising: receiving a training input audio sequence (400) including a sequence of input frames, the sequence of input frames defining a hot word that initiates a wake-up process on a user device (102); obtaining a first label (420, 420a) for the training input audio sequence; obtaining a second label (420, 420b) for the training input audio sequence, the second label (420, 420b) being different from the first label (420, 420a); using the stored neural network (300) and the training input audio sequence (400) to generate an output (350) indicative of the likelihood that the training input audio sequence (400) contains the hot word; determining a first loss (710, 710a) based on the first label (420, 420a) and the output (350); determining a second loss (710, 710b) based on the second label (420, 420b) and the output (350); optimizing the stored neural network (300) based on the first loss (710, 710a) and the second loss (710, 710b) associated with the training input audio sequence (400); The method (800).

2. 2. The method of claim 1, wherein the stored neural network includes an encoder and a decoder, each of the encoder and the decoder of the stored neural network including sequentially stacked singular value decomposition filter (SVDF) layers.

3. 3. The method (800) of claim 1 or 2, wherein the output (350) is based on a probability distribution of logits (502) based on the training input audio sequences (400).

4. The method (800) of any of claims 1 to 3, wherein said operations further comprise smoothing the logits (502) before determining the first loss (710, 710a).

5. Determining the first loss (710, 710a) generating a plurality of encoder windows (310w), each encoder window of the plurality of encoder windows (310w) associated with one or more phonemes of the hot word; determining the first loss (710, 710a) for each encoder window of the plurality of encoder windows (310w); The method (800) of any of claims 1 to 4, comprising:

6. The method (800) of any of claims 1 to 5, wherein a collection size of the plurality of encoder windows (310w) corresponds to an average acoustic length of the hot words.

7. The method (800) of any of claims 1 to 6, wherein optimizing the stored neural network (300) comprises generating a weighted average of the first loss (710, 710a) and the second loss (710, 710b).

8. The method (800) of any of claims 1 to 7, wherein the second label (420, 420b) is derived from one or more phoneme sequences of the hot word.

9. The method (800) of any of claims 1 to 8, wherein the first label (420, 420a) is based on the position of the last phoneme of the hot word.

10. The method (800) according to any of the preceding claims, wherein the sequence of input frames each comprises one or more respective audio features (410) characterizing phonetic components (430) of the hot words.

11. the first label (420, 420a) comprises a max pooling loss label; the second label (420, 420b) comprises a cross-entropy label; the first loss (710, 710a) comprises a max pooling loss; the second loss (710, 710b) comprises a cross-entropy loss; The method (800) according to any of the preceding claims.

12. A system (100), comprising: Data processing hardware (103, 112); and memory hardware (105, 114) in communication with said data processing hardware (103, 112), said memory hardware (105, 114) storing instructions that, when executed on said data processing hardware (103, 112), cause said data processing hardware (103, 112) to perform operations, said operations being: receiving a training input audio sequence (400) including a sequence of input frames, the sequence of input frames defining a hot word that initiates a wake-up process on a user device (102); obtaining a first label (420, 420a) for the training input audio sequence; obtaining a second label (420, 420b) for the training input audio sequence, the second label (420, 420b) being different from the first label (420, 420a); using the stored neural network (300) and the training input audio sequence (400) to generate an output (350) indicative of the likelihood that the training input audio sequence (400) contains the hot word; determining a first loss (710, 710a) based on the first label (420, 420a) and the output (350); determining a second loss (710, 710b) based on the second label (420, 420b) and the output (350); optimizing the stored neural network (300) based on the first loss (710, 710a) and the second loss (710, 710b) associated with the training input audio sequence; The system (100).

13. 13. The system of claim 12, wherein the stored neural network includes an encoder and a decoder, each of the encoder and the decoder of the stored neural network including sequentially stacked singular value decomposition filter (SVDF) layers.

14. 14. The system (100) of claim 12 or 13, wherein the output (350) is based on a probability distribution of logits (502) based on the training input audio sequences (400).

15. The system (100) of any of claims 12 to 14, wherein the operations further comprise smoothing the logits (502) before determining the first loss (710, 710a).

16. Determining the first loss (710, 710a) generating a plurality of encoder windows (310w), each encoder window of the plurality of encoder windows (310w) associated with one or more phonemes of the hot word; determining the first loss (710, 710a) for each encoder window of the plurality of encoder windows (310w); The system (100) according to any one of claims 12 to 15, comprising:

17. The system (100) of any of claims 12 to 16, wherein a collective size of the plurality of encoder windows (310w) corresponds to an average acoustic length of the hot words.

18. 18. The system (100) of any of claims 12 to 17, wherein optimizing the stored neural network (300) comprises generating a weighted average of the first loss (710, 710a) and the second loss (710, 710b).

19. The system (100) of any of claims 12 to 18, wherein the second label (420, 420b) is derived from one or more phoneme sequences of the hot word.

20. The system (100) of any of claims 12 to 19, wherein the first label (420, 420a) is based on the position of the last phoneme of the hot word.

21. The system (100) of any of claims 12 to 20, wherein the sequence of input frames each comprises one or more respective audio features (410) characterizing phonetic components (430) of the hot words.

22. the first label (420, 420a) comprises a max pooling loss label; the second label (420, 420b) comprises a cross-entropy label; the first loss (710, 710a) comprises a max pooling loss; the second loss (710, 710b) comprises a cross-entropy loss; A system (100) according to any one of claims 12 to 21.

Images