Mixing heterogeneous loss types to improve keyword spotting accuracy

Abstract

A method (800) for training a stored neural network (300) includes receiving a training input audio sequence (400) including a sequence of input frames defining a hot word that initiates a wake-up process on a user device (102). The method further includes obtaining a first label (420, 420a) and a second label (420, 420b) for the training input audio sequence. The method includes using the stored neural network and the training input audio sequence to generate an output (350) indicative of a likelihood that the training input audio sequence includes the hot word. The method further includes determining a first loss (710, 710a) based on the first label and the output. The method includes determining a second loss (710, 710b) based on the second label and the output. The method further includes optimizing the stored neural network based on the first loss and the second loss associated with the training input audio sequence.

Description

[Technical Field]

[0001] This disclosure relates to a system for keyword spotting in streaming audio. [Background technology]

[0002] In voice-enabled environments (e.g., homes, workplaces, schools, cars), users can speak queries or commands aloud to computer-based systems that process queries, respond, and / or perform functions based on commands. Voice-enabled environments can be implemented using a network of connected microphone devices distributed throughout various rooms or areas of the environment. These devices may use so-called "hotwords" to help identify when a given utterance is directed at the system, as opposed to utterances directed at other individuals present in the environment. Therefore, devices can operate in sleep or hibernation states and only wake up if the detected utterance contains a hotword. For a voice-enabled environment to function optimally, devices within the environment must be able to accurately and efficiently detect hotwords. Neural networks have recently emerged as an attractive solution for training models to detect user-spoken hotwords within streaming audio. [Overview of the Initiative]

[0003] One aspect of this disclosure provides a method for training a hotword detector using two labels and two loss functions for training data. When executed on data processing hardware, the computer implementation causes the data processing hardware to perform an operation that includes receiving a training input audio sequence containing a sequence of input frames. The sequence of input frames defines a hotword that initiates a wake-up process on a user device. The operation includes obtaining a first label and a second label for the training input audio sequence, wherein the second label is different from the first label. The operation also includes using a stored neural network and the training input audio sequence to generate an output indicating the likelihood that the training input audio sequence contains a hotword. The operation includes determining a first loss based on the first label and the output, and a second loss based on the second label and the output. The operation further includes optimizing the stored neural network based on the first and second losses associated with the training input audio sequence.

[0004] Embodiments of this disclosure may include one or more of the following optional features. In some embodiments, the memorized neural network includes an encoder and a decoder, each of which includes sequentially stacked single-value decomposition filter (SVDF) layers. In other embodiments, the output is based on a probability distribution of logits based on a training input audio sequence. In these embodiments, the operation further includes smoothing the logits before determining a first loss.

[0005] In some additional embodiments, determining the first loss includes generating a plurality of encoder windows, where each encoder window of the plurality of encoder windows is associated with one or more phonemes of the hotword. These embodiments further include determining the first loss for each encoder window of the plurality of encoder windows. In these embodiments, the aggregate size of the plurality of encoder windows may correspond to the average acoustic length of the hotword.

[0006] Furthermore, optimizing the memorized neural network may involve generating a weighted average of the first and second losses. In some embodiments, the second label is derived from a sequence of one or more phonemes of the hotword. Alternatively, the first label may be based on the position of the last phoneme of the hotword. In addition, each sequence of input frames may contain one or more respective audio features that characterize the speech components of the hotword.

[0007] In some exemplary embodiments, the first label is the maximum pooling loss label, and the second label is the cross-entropy label. In these exemplary embodiments, the first loss is the maximum pooling loss, and the second loss is the cross-entropy loss.

[0008] Another aspect of this disclosure provides a system for training a hotword detector using two labels and two loss functions for training data. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform an operation which includes receiving a training input audio sequence which includes a sequence of input frames. The sequence of input frames defines a hotword which initiates a wake-up process on a user device. The operation is to obtain a first label and a second label for the training input audio sequence which includes obtaining a second label which is different from the first label. The operation also includes using a stored neural network and the training input audio sequence to generate an output which indicates the likelihood that the training input audio sequence contains a hotword. The operation is to determine a first loss based on the first label and the output, and a second loss based on the second label and the output. The operation is to further include optimizing the stored neural network based on the first and second losses associated with the training input audio sequence.

[0009] This embodiment may include one or more of the following optional features. In some embodiments, the memorized neural network includes an encoder and a decoder, each of which includes sequentially stacked singular value decomposition (SVDF) layers. In other embodiments, the output is based on a probability distribution of logits based on a training input audio sequence. In these embodiments, the operation further includes smoothing the logits before determining a first loss.

[0010] In some additional embodiments, determining the first loss includes generating a plurality of encoder windows, where each encoder window of the plurality of encoder windows is associated with one or more phonemes of the hotword. These embodiments further include determining the first loss for each encoder window of the plurality of encoder windows. In these embodiments, the aggregate size of the plurality of encoder windows may correspond to the average acoustic length of the hotword.

[0011] Furthermore, optimizing the memorized neural network may involve generating a weighted average of the first and second losses. In some embodiments, the second label is derived from a sequence of one or more phonemes of the hotword. Alternatively, the first label may be based on the position of the last phoneme of the hotword. In addition, each sequence of input frames may contain one or more respective audio features that characterize the speech components of the hotword.

[0012] In some exemplary embodiments, the first label is the maximum pooling loss label, and the second label is the cross-entropy label. In these exemplary embodiments, the first loss is the maximum pooling loss, and the second loss is the cross-entropy loss.

[0013] Details of one or more embodiments of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings, as well as from the claims. [Brief explanation of the drawing]

[0014] [Figure 1] This is a schematic diagram of an exemplary system for training a memorized neural network and for detecting hotwords in spoken utterances using the trained memorized neural network. [Figure 2]Schematic diagram of components of a typical neural network acoustic encoder used by a model for detecting hotwords. [Figure 3A] Schematic diagram of exemplary components of the memory neural network of the system of FIG. 1. [Figure 3B] Schematic diagram of exemplary components of a memory neural network having multiple layers. [Figure 4A] Schematic diagram showing a pair of audio features and labels generated from streaming audio for training a neural network. [Figure 4B] Schematic diagram showing a pair of audio features and labels generated from streaming audio for training a neural network. [Figure 5A] Schematic diagram of the layers of the memory neural network of the system of FIG. 1. [Figure 5B] Schematic diagram of the layers of the memory neural network of the system of FIG. 1. [Figure 5C] Schematic diagram of an exemplary training process for the memory neural network of the system of FIG. 1. [Figure 6] Graphic representation of an example of a window used during the training process of FIG. 5C. [Figure 7] Schematic diagram of an exemplary training process of the memory neural network of FIG. 1 using two labels and two loss functions. [Figure 8] Flowchart of an exemplary configuration of operations of a method for training a neural network to detect hotwords. [Figure 9] Schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.

MODE FOR CARRYING OUT THE INVENTION

[0015] Like reference symbols in the various drawings refer to like elements.

[0016] Voice-enabled devices (e.g., user devices running voice assistants) enable users to speak queries or commands aloud, process and respond to queries, and / or perform functions based on commands. The use of “hotwords” (also known as “keywords,” “attention words,” “wake-up phrases / words,” “trigger phrases,” or “voice action initiation commands”), where predetermined terms / phrases are reserved by convention to draw the attention of the voice-enabled device, allows the device to distinguish between utterances directed at the system (i.e., those intended to initiate a wake-up process to process one or more terms following a hotword in the utterance) and utterances directed at individuals in the environment. Typically, voice-enabled devices operate in sleep mode to conserve battery power and do not process input audio data unless there is input audio data following a spoken hotword. For example, while in sleep mode, a voice-enabled device captures input audio via a microphone and uses a hotword detector trained to detect the presence of a hotword in the input audio. When a hotword is detected in the input audio, the voice-enabled device initiates a wake-up process to process the hotword and / or any other terms in the input audio following the hotword.

[0017] Hotword detection is like finding a needle in a haystack. This is because the hotword detector must continuously listen to streaming audio and trigger precisely and instantaneously when the presence of a hotword is detected within the streaming audio. In other words, the hotword detector is tasked with ignoring the streaming audio unless the presence of a hotword is detected. Generally, neural networks are used by hotword detectors to cope with the complexity of detecting the presence of a hotword in a continuous stream of audio.

[0018] A hotword detector typically includes three main components: a signal processing front-end, a neural network acoustic encoder, and a manually designed decoder. The signal processing front-end can convert the raw audio signal captured by the user device's microphone into one or more audio features formatted for processing by the neural network acoustic encoder component. For example, the neural network acoustic encoder component may convert these audio features into phonemes, and the manually designed decoder, using a hand-coded algorithm, stitches the phonemes together to provide a probability of whether the audio sequence contains a hotword.

[0019] A common method for training a neural network involves providing it with labeled training samples. These training samples are typically pre-screened data inputs labeled based on the desired output of the neural network. For example, in a hotword detector, the training samples are labeled with an indication of the presence of a hotword (e.g., "1" if the hotword is present in the training samples, "0" otherwise). The neural network analyzes the training samples and then generates outputs or predictions that are compared to a predefined target output (i.e., labels) to determine a loss using a loss function. The loss represents the accuracy of the output compared to the labels. The loss is then fed back into the neural network, which adjusts one or more weights, values, or parameters based on the loss.

[0020] When training a hotword detector, training samples may include audio sequences, and the neural network may output an indication or probability that the audio sequence contains a hotword. While this training process may seem straightforward, many different ways of labeling audio sequences exist, and each labeling convention results in a unique loss. Similarly, each resulting loss tunes the hotword detector in a specific way. For example, a simple cross-entropy-based loss uses labels derived from phoneme sequences. Cross-entropy-based losses are highly susceptible to the positional alignment of all subphonemes containing the keyword, and therefore the resulting trained hotword detector uses a significant portion of its resources to learn to predict precise positions in order to signal pattern detection. In another example, the max-pooling technique uses labels derived only from the position of the last phoneme of the keyword. Thus, unlike the cross-entropy example above, the max-pooling technique does not depend on the positional information of other subphonemes, and the resulting hotword detector can ignore noise in positional alignment, making it more stable than a hotword detector trained using cross-entropy-based losses. However, each of the above examples has its own advantages and disadvantages when compared to others or to other technologies.

[0021] Embodiments herein relate to an end-to-end hotword spotting system (also known as a “keyword spotting system”) that trains a hotword detector using two distinct target labels per sample in a training input set. Similarly, for each training sample, the two target labels are each used to determine two distinct losses. The hotword detector then receives losses based on both of these losses and adjusts accordingly. By using two losses determined differently using two different labels, the hotword detector can be fine-tuned using multiple techniques. For example, training a hotword detector using loss functions based on cross-entropy and maximum pooling simultaneously can yield a hotword detector that is more accurate and efficient than one trained using either technique individually.

[0022] Referring to Figure 1, in some embodiments, the exemplary system 100 includes one or more user devices 102, each associated with a respective user 10, and communicating with a remote system 110 via a network 104. Each user device 102 may correspond to a computing device such as a mobile phone, computer, wearable device, smart appliance, or smart speaker, and comprises data processing hardware 103 and memory hardware 105. The remote system 110 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources 112 (e.g., data processing hardware) and / or storage resources 114 (e.g., memory hardware). The user device 102 receives a trained memorized neural network 300 from the remote system 110 via the network 104 and runs the trained memorized neural network 300 to detect hotwords in the streaming audio 118. The trained and memorized neural network 300 may reside in the hotword detector 106 (also called a hotworder) of the user device 102, which is configured to detect the presence of hotwords in the streaming audio without performing semantic analysis or speech recognition processing on the streaming audio 118. Optionally, the trained and memorized neural network 300 may also reside in the automatic speech recognition (ASR) 108 of the user device 102 and / or the remote system 110 to verify that the hotword detector 106 has correctly detected the presence of hotwords in the streaming audio 118.

[0023] In some embodiments, the data processing hardware 112 trains a memorized neural network 300 using training samples 400 obtained from an annotated speech pool 130. The training samples may include a first label 420, 420a and a second label 420, 420b. That is, each training sample may be annotated with two distinct labels 420a, 420b. The annotated speech pool 130 may reside in the memory hardware 114 and / or some other remote memory location(s). In the illustrated example, when user 10 speaks an utterance 120 that includes a hotword (e.g., "Hey Google") captured as streaming audio 118 by user device 102, a memorized neural network 300 running on user device 102 is configured to detect the presence of the hotword in the utterance 120 so that user device 102 initiates a wake-up process to process the hotword in the utterance 120 and / or one or more other terms (e.g., a query or command) that follow the hotword in the utterance 120. In an additional embodiment, user device 102 sends the utterance 120 to a remote system 110 for additional processing or verification (e.g., using another, potentially more computationally intensive memorized neural network 300).

[0024] In the illustrated example, the memorized neural network 300 includes an encoder section 310 and a decoder section 311, each containing a layered topology of singular value decomposition (SVDF) layers 302. The SVDF layers 302 provide memory to the neural network 300 by providing memory capacity to each SVDF layer 302 such that the memory capacity of all SVDF layers 302 additionally constitutes overall fixed memory for the neural network 300 to store only the fixed length of time of streaming audio 118 required to capture the audio features 410 (Figures 4A and 4B) that characterize the hotwords. It should be understood that this architecture of the memorized neural network 300 is illustrative and any other architecture of the memorized neural network 300 may be substituted.

[0025] In some embodiments, the memorized neural network 300 is trained using multiple labels 420, 420a-b to generate corresponding losses 710, 710a-b for each of the corresponding labels 420a-b. The process of training the neural network 300 using multiple labels 420 is described in more detail below (Figure 7).

[0026] Referring to Figure 2, a typical hotword detector uses a memoryless neural network acoustic encoder 200. Because the network 200 has no memory, each neuron 212 of the acoustic encoder 200 must simultaneously accept as input every audio feature from every frame 210, 210a-d of the spoken utterance 120. Note that each frame 210 can have any number of audio features, and each of these audio features is accepted as input by neuron 212. Such a configuration requires a considerably large neural network acoustic encoder 200, which increases dramatically as the fixed time duration increases and / or as the number of audio features increases. The output of the acoustic encoder 200 results in, for example, the probability of each phoneme of the detected hotword. The acoustic encoder 200 then must process its output (e.g., by piecing together the phonemes) in reliance on a hand-coded decoder to generate a score (i.e., an estimate) indicating the presence of the hotword.

[0027] Referring here to Figures 3A and 3B, in some embodiments, a singular value decomposition (SVDF) neural network 300 (also called a memorized neural network) has any number of neurons / nodes 312, and each neuron 312 accepts only a single frame 210, 210a-d of spoken utterances 120 at a time. That is, if each frame 210 constitutes, for example, 30ms of audio data, then each frame 210 is input to neuron 312 approximately every 30ms (i.e., time 1, time 2, time 3, time 4, etc.). Figure 3A shows each neuron 312 including a two-stage filtering mechanism, a first stage 320 (i.e., stage 1 feature filter) that performs filtering on the feature dimension of the input, and a second stage 340 (i.e., stage 2 time filter) that performs filtering on the time dimension of the output of the first stage 320. Thus, the stage 1 feature filter 320 performs feature filtering only on the current frame 210. Next, the processing results are placed in the memory component 330. In these examples, the size of the memory component 330 can be configured per node level or per layer level. After the Stage 1 feature filter 320 processes a given frame 210 (for example, by filtering audio features within the frame), the filtered results are placed in the next available memory locations 332, 332a-d within the memory component 330. When all memory locations 332 are full, the Stage 1 feature filter 320 overwrites the memory location 332 in the memory component 330 that stores the oldest filtered data. For illustrative purposes, Figure 3A shows a memory component 330 of size 4 (four memory locations 332a-d) and four frames 210a-d, but note that due to the nature of hotword detection, the system 100 typically continuously monitors the streaming audio 118 so that each neuron 312 "slides" along the pipeline or processes the frames 210 in a similar manner to the pipeline.In other words, if each stage contains N feature filters 320 and N time filters 340 (each matching the size of the input feature frame 210), the layer is analogous to computing an N × T (where T is equal to the number of frames 210 within a fixed period) convolution of the feature filters by sliding each of the N filters 320, 340 on the input feature frame 210 with a stride of the size of the feature frame. For example, the example shows the memory component 330 in capacity after the feature filter of stage 1 outputs the filtered audio features associated with frame 4 (F4) 210d (during time 4), so the feature filter 320 of stage 1 places the filtered audio features associated with the subsequent frame 5 (F5) (during time 5) into memory 330 by overwriting the filtered audio features associated with frame 1 (F1) 210a in memory location 332a. In this way, the Stage 2 time filter 340 applies filtering to the previous T-1 (where T is again equal to the number of frames 210 within a fixed period) filtered audio features output from the Stage 1 feature filter 320.

[0028] Next, the Stage 2 time filter 340 filters each filtered audio feature stored in memory 330. For example, Figure 3A shows that each time the Stage 1 feature filter 320 stores a new filtered audio feature in memory 330, the Stage 2 time filter 340 filters the audio features in each of the four memory locations 332. In this way, the Stage 2 time filter 340 is always filtering several past frames 210, the number of which is proportional to the size of memory 330. Each neuron 312 is part of a single SVDF layer 302, and the neural network 300 may contain any number of layers 302. The output of each Stage 2 time filter 340 is passed to the input of a neuron 312 in the next layer 302. The number of layers 302 and the number of neurons 312 per layer 302 are fully configurable and depend on the available resources as well as the desired size, power, and precision. This disclosure is not limited to the number of SVDF layers 302 or the number of neurons 312 within each SVDF layer 302.

[0029] Referring now to Figure 3B, each SVDF layer 302, 302a~n (or simply “layers”) of the neural network 300 is connected such that, in some embodiments, the output of the previous layer is accepted as input to the corresponding layer 302. In some examples, the final layer 302n outputs a probability score 350 indicating the probability that the utterance 120 contains a hotword.

[0030] In the illustrated example SVDF network 300, the layer design derives from the concept that a densely connected layer 302 processing a sequence of input frames 210 can be approximated by using the singular value decomposition of each of its nodes 312. The approximation is configurable. For example, rank R approximation means extending a new dimension R to the layer's filter, where stage 1 occurs independently and stage 2 adds up the outputs of all ranks before passing through the nonlinearity. In other words, the SVDF decomposition of the nodes 312 of a densely connected layer of matching dimensions may be used to initialize the SVDF layer 302, which results in a principal initialization and improves the quality of the layer's generalization. Essentially, the "power" of a larger densely connected layer is potentially transferred to a much smaller SVDF (depending on the rank). However, it should be noted that the SVDF layer 302 does not require initialization to outperform even densely connected or convolutional layers with the same or even more operations.

[0031] In some embodiments, the system 100 includes a stateful, stackable neural network 300, where each neuron 312 in each SVDF layer 302 includes a first stage 320 associated with filtering audio features and a second stage 340 associated with filtering the output of the first stage 320 with respect to time. Specifically, the first stage 320 is configured to perform filtering on one or more audio features on one audio feature input frame 210 at a time and output the filtered audio features to their respective memory components 330. Here, the stage 1 feature filter 320 receives one or more audio features associated with the time frame 210 as input for processing and outputs the processed audio features to their respective memory components 330 of the SVDF layer 302. The second stage 340 is then configured to perform filtering on all filtered audio features output from the first stage 320 and present in their respective memory components 330. For example, if each memory component 330 is equal to 8, the second stage 340 will pull up to 8 of the last filtered audio features present in the memory component 330 output from the first stage 320 during the individual filtering of audio features in the sequence of 8 input frames 210. When the first stage 320 fills the corresponding memory component 330 to its capacity, the memory location 332 containing the oldest filtered audio feature is overwritten (i.e., first-in, first-out). Thus, depending on the capacity of the memory component 330 in the SVDF neuron 312 or layer 302, the second stage 340 can store some past outputs processed by the first stage 320 of the corresponding SVDF layer 302. Furthermore, since the memory component 330 in the SVDF layer 302 is additive, each SVDF neuron 312 and the memory component 330 in layer 302 also includes the memory of the preceding SVDF neuron 312 and layer 302, and thus the overall receptive field of the memorized neural network 300 is expanded.For example, in a neural network 300 topology having four SVDF layers 302, each having a single neuron 312 with a memory component 330 equal to 8, the last SVDF layer 302 contains a sequence of up to 32 last audio feature input frames 210 that have been individually filtered by the neural network 300. However, it should be noted that the amount of memory can be configured per layer 302 or per node 312. For example, the first layer 302a may be allocated 32 positions 332, and the last layer 302 may consist of 8 positions 332. As a result, the stacked SVDF layers 302 allow the neural network 300 to process only one input time frame 210 (e.g., 30 milliseconds of audio data) at a time, and to incorporate several filtered audio features in the past that capture a fixed duration necessary to capture a specified hotword in the streaming audio 118. In contrast, a memory-less neural network 200 (as shown in Figure 2) requires its neurons 212 to process all audio feature frames covering a fixed duration (e.g., 2 seconds of audio data) at once to determine the probability that streaming audio contains the presence of a hotword. This dramatically increases the overall size of the network. Furthermore, while recurrent neural networks (RNNs) using long short-term memory (LSTM) provide memory, RNN-LSTMs cause neurons with virtually infinite memory to continuously update their state after each processing instance, thereby hindering their ability to remember a finite number of past processed outputs, in which case each new output overwrites the previous one (when the fixed-size memory is full). In other words, an SVDF network does not regress outputs to a state (memory) and does not rewrite all states with each iteration. Instead, memory maintains the state of each estimated execution, isolated from subsequent executions, rather than pushing and popping new entries based on the memory size configured for the layer.

[0032] Referring here to Figures 4A and 4B, in some embodiments, the memorized neural network 300 is trained on a plurality of training input audio sequences 400 (i.e., training samples), each containing an input frame 210, a sequence of 210a-n, and two or more labels 420a-b assigned to the input frame 210. Each input frame 210 contains one or more respective audio features 410 that characterize the speech components 430 of the hotword, and each label 420 indicates the probability that one or more audio features 410 of each input frame 210 contain the speech components 430 of the hotword. In some examples, the audio features 410 of each input frame 210 are transformed from the raw audio signal 402 of the audio stream 118 during a preprocessing step 404. The audio features 410 may include one or more log filter banks. Therefore, the preprocessing step may segment the audio stream 118 (or spoken utterance 120) into a sequence of input frames 210 (e.g., each 30 ms), generating a separate log filter bank for each frame 210. For example, each frame 210 may be represented by 40 log filter banks. Furthermore, each successive SVDF layer 302 receives, as input, time-filtered audio features 410 output from the previous SVDF layer 302.

[0033] In the illustrated example, each training input audio sequence 400 is associated with a training sample containing annotated utterances (i.e., having labels 420a-b) that include a specified hotword occurring within a fixed time (e.g., 2 seconds). The memorized neural network 300 may also be optionally trained against annotated utterances 400 that do not contain the specified hotword, or that contain the specified hotword but span a longer time than the fixed time, so that the fixed memory does not falsely detect data outside the fixed time. In some examples, the fixed time corresponds to the time it takes a typical speaker to say the specified hotword to invoke the user device 102 to process the spoken query and / or voice command. For example, if the specified hotword includes the phrase "Hey Google" or "Ok Google," a fixed time set to 2 seconds would likely be sufficient, since even a slow speaker would generally not take more than 2 seconds to say the specified phrase. Therefore, since the only important thing is to detect the occurrence of a specified hotword in the streaming audio 118 over a fixed time period, the neural network 300 includes a fixed amount of fixed memory proportional to the amount of audio over the fixed time period (e.g., 2 seconds). Thus, the fixed memory of the neural network 300 allows the neurons 312 of the neural network to store the most recent filtered audio features 410 over the fixed time period while filtering audio features 410 (e.g., a log filter bank) from one input frame 210 of the streaming audio 118 at a time (e.g., a 30ms time window), and to remove or delete any filtered audio features 410 outside the fixed time period from the current filtering iteration.Therefore, if the neural network 300 has a memory depth of, for example, 32, the first 32 frames processed by the neural network 300 will fill the memory component 330 to its capacity, and after the first 32, for each new output, the neural network 300 will remove the oldest processed audio feature from the corresponding memory location 332 of the memory component 330.

[0034] Referring to Figure 4A, in the case of end-to-end training, the training input audio sequence 400a contains labels 420a that can be applied to each input frame 210. In some examples, if the training sample 400a contains a hotword, the target label 420a associated with the target score (e.g., "1") is applied to one or more input frames 210 that contain audio features 410 that characterize the last or near-last audio component 430 of the hotword. For example, if the audio component 430 of the hotword "OK Google" is divided into "ou", "k", "eI", "<silence>", "g", "u", "g", "@", and "l", then the target label of number "1" is applied to all input frames 210 corresponding to the letter "l" (i.e., the last component 430 of the hotword), which are part of the required sequence of the audio component 430 of the hotword. In this scenario, all other input frames 210 (not associated with the last audio component 430) are assigned different labels (e.g., "0"). Therefore, each input frame 210 contains a corresponding pair of input features and labels 410, 420a. The input features 410 are typically one-dimensional tensors computed from the input audio across the input frames 210, corresponding to, for example, a Mel filter bank or a log filter bank.

[0035] The exemplary label 420a focuses on the position of the last phoneme of the hotword and does not depend on the positional information of other subphonemes (and therefore, label "0" for speech components that are not "l"). Typically, this type of label 420a is associated with the maximum pooling loss, which does not depend on the exact position of the target pattern and attempts to define the presence of the pattern within a defined interval. Labels 420a are generated from the annotated utterance 400a, and each input feature tensor 410 is assigned a speech class by a forced alignment step (i.e., label "1" is given to pairs corresponding to the last class belonging to the hotword, and "0" is given to all the rest). Thus, the training input audio sequence 400a contains binary labels assigned to the sequence of input frames. The annotated utterance 400a, or training input audio sequence 400a, corresponds to the training sample 400 taken from the annotated utterance pool 130 in Figure 1.

[0036] In another example, Figure 4B includes a training input audio sequence 400b, which includes labels 420b associated with a score that increases along the sequence of input frames 210 as the number of audio features 410 that characterize (match) the speech components 430 of the hotword increases. For example, if the hotword includes "Ok Google", the input frame 210 containing the respective audio features 410 characterizing the first speech components "o" and "k" is assigned the label 420b of "1", the input frame 210 containing the respective audio features 410 characterizing the last speech component "l" is assigned the label 420b of "5", and the input frame 210 containing the respective audio features 410 characterizing the intermediate speech components 430 is assigned the labels 420b of "2", "3", and "4".

[0037] In additional embodiments, the number of positive labels 420b is increased. For example, a fixed number of "1" labels 420b are generated starting from a first frame 210 containing an audio feature 410 characterizing the last audio component 430 of the hotword. In this implementation, when the configured number of positive labels 420b (e.g., "1") is large, positive labels 420b may be applied to frames 210 to which otherwise non-positive labels 420b (e.g., "0") would have been applied. In other examples, the starting position of the positive labels 420b is modified. For example, labels 420b may be shifted to start at the beginning, middle, or end of the segment of frame 210 containing the last keyword audio component 430. In yet another example, weight loss is associated with the input sequence. For example, weight loss data is added to the input sequence, allowing the training procedure to reduce losses (i.e., error gradients) caused by small misalignments. Specifically, in frame-based loss functions, losses may arise from either misclassification or misalignment. To reduce losses, the neural network 300 predicts both the correct label 420b and the correct position (timing) of label 420b. Even if the network 300 detects a keyword at some point, the result may be considered erroneous if it is not perfectly aligned with a given target label 420b. Therefore, loss weighting is particularly useful for frames 210, where the likelihood of misalignment during the forced alignment phase is high. The exemplary label 420b is typically related to cross-entropy loss, resulting in a model that is highly susceptible to the positional alignment of all subphonemes of the keyword.

[0038] As a result of training using either of the training input audio sequences 400a or 400b in Figures 4A-4B, the neural network 300 is optimized (using the determined loss) to produce an output 350 indicating whether a hotword(s) is present in the streaming audio 118. In some examples, the network 300 is trained in two stages. Referring here to Figure 5A, schematic Figure 500a shows the encoder portion (or simply the “encoder”) 310a of the neural network 300, which includes, for example, eight layers individually trained to produce acoustic posterior probabilities. In addition to the SVDF layer, the network 300 may include, for example, bottleneck, softmax, and / or other layers. To train the encoder 310a, label generation assigns a separate class to all speech components of the hotword (plus an “epsilon” target for silence and everything that is not a hotword). Next, the decoder portion (or simply "decoder") 311a of the neural network 300 is trained by creating a topology in which the first portion (i.e., layers and connections) matches the topology of the encoder 310a, and a checkpoint selected from that encoder 310a of the neural network 300 is used to initialize the neural network 300. The training is specified to "freeze" (i.e., not update) the parameters of the encoder 310a, and thus only the decoder 311a portion of the topology is adjusted. This naturally produces a single spotter neural network, even if it is the product of two staggered training pipelines. Training using this method is particularly useful for models that tend to overfit to parts of the training set.

[0039] Alternatively, the neural network 300 is trained end-to-end from the start. For example, the neural network 300 directly accepts features (similar to the training of the encoder 310a described above), but instead uses the output of a binary target label 420a (i.e., "0" or "1") for use in training the decoder 311a. Such an end-to-end neural network 300 can use any topology. For example, schematic Figure 500b, as shown in Figure 5B, shows a topology for the neural network 300 with encoder 310b and decoder 311b, similar to the topology in Figure 5A except that encoder 310b does not include an intermediate softmax layer. Similar to the topology in Figure 5A, the topology in Figure 5B can use a pre-trained encoder checkpoint, which has a precision to adjust how the decoder 311b portion is tuned (for example, if the precision is set to 0, it is equivalent to the topology in Figure 5A). This end-to-end pipeline tends to outperform the separately trained encoder 310a and decoder 311a in Figure 5A, especially for smaller models that do not tend to overfit, when the entire topology parameter is tuned.

[0040] Therefore, the neural network 300 can avoid using a manually tuned decoder. Manually tuning a decoder increases the difficulty of changing or adding hotwords. A single memorized neural network 300 can be trained to detect multiple different hotwords, as well as the same hotword spanning two or more locations. Furthermore, detection quality is potentially lower compared to a network specifically optimized for hotword detection, trained with millions of examples. Moreover, a typical manually tuned decoder is more complex than a single neural network that performs both encoding and decoding. Conventional systems tend to be over-parameterized, consuming significantly more memory and computation than comparable end-to-end models, and failing to fully utilize neural network acceleration hardware. Furthermore, manually tuned decoders have problems with accented speech, thereby making it extremely difficult to create a detector that can function across multiple regions and / or languages.

[0041] The memorized neural network 300 outperforms a simple fully connected layer of the same size, but also benefits from the ability to arbitrarily initialize parameters from a pre-trained fully connected layer. Network 300 allows for finer control over the amount of information recalled from the past. This means that for certain tasks where paying attention to theoretically infinite past information (e.g., continuously listening to streaming audio) is not beneficial (and actually detrimental), RNN-LSTM will outperform. However, Network 300 can work in conjunction with RNN-LSTM, typically by utilizing SVDFs in the lower layers to filter out noisy, low-level features from the past and leveraging LSTMs in the upper layers. Assuming several relatively small filters include SVDFs, the number of parameters and computations are precisely controlled. This is helpful when choosing the trade-off between quality and size / computation. Furthermore, because of this quality, Network 300 allows for the creation of very small networks that perform better than other topologies, such as simple convolutional neural networks (CNNs) operating at a larger granularity.

[0042] Referring to Figures 5C and 6, in some configurations, the neural network 300 is optimized using smoothed maximum pooling loss, rather than optimizing the neural network 300 to generate a probability score 350 indicating the likelihood that a hotword(s) is present in the streaming audio 118 using cross-entropy loss. Here, as in the example shown in Figures 5A and 5B, this method involves jointly training encoders 310, 310c and decoders 311, 311c. With this smoothed maximum pooling loss method, the neural network 300 can be trained to detect not only a portion of a hotword (e.g., using encoder 310c) but also the entire hotword (e.g., using decoder 311c). By using the smoothed maximum pooling loss method, this method is independent of frame labels 420a~b and may be suitable for embodiments such as on-device learning (e.g., for user device 102).

[0043] In hotword detection, the precise location of the hotword is generally less important than the actual presence of the hotword. Therefore, alignment of frame labels 420 can cause hotword detection errors (i.e., hotword detection is potentially impaired). This alignment can be particularly problematic when frame labels 420 have inherent uncertainties caused by noise or the accent of a particular word. In the case of frame labels 420, the training input audio sequence 400 often contains intervals of similar or identical frame labels 420 that are repeated, called runs. For example, both Figures 4A and 4B contain runs of "0". These runs indicate that when training the network 300, the network 300 should make strong learning associations for the generation of output 350. In contrast, smoothed max pooling techniques (e.g., shown in Figures 5C and 6) avoid using frame labels 420 to specify precise activation locations (i.e., timings).

[0044] In the smoothed max pooling loss approach, in some examples, an initial loss is defined for both the encoder 310c and the decoder 311c, and then the initial losses of the encoder 310c and the decoder 311c are optimized simultaneously. Max pooling refers to a sample-based discretization process in which the dimensionality of an input is reduced by applying a max filter. In some examples, the training process 500c using the smoothed max pooling technique includes smoothing operations 510, 510e~d and max pooling operations 520, 520e~d. In these examples, the smoothing operation 510 occurs before the max pooling operation 520. Here, during the smoothing operation 510, the training process 500c performs time smoothing on frame 210. For example, the training process 500c smooths the logits 502, 502e~d corresponding to frame 210. A logit generally refers to a vector or other raw prediction form output from one or more SVDF layers 302. The logit 502 serves as input to the softmax portion of the encoder 310 and / or decoder 311, so that the encoder 310 and / or decoder 311 generate output probabilities based on the input of one or more logits 502. For example, the logit 502 is an unnormalized prediction data format, and the softmax normalizes the logit 502 to probabilities (e.g., the probability of a hot word).

[0045] By having a smoothing operation 510 before the max pooling operation 520, the training process 500c trains the network 300 more stably to small fluctuations and temporal shifts within the streaming audio 118. This high stability is in contrast to other training techniques that may use some form of max pooling operation without a temporal smoothing operation. For example, other training techniques may use max pooling in the time domain and determine the cross-entropy loss with respect to the logit 502 of the frame 210 having the maximum activation. By introducing a temporal smoothing operation 510 before the max pooling operation 520, the training process 500c of the network 300 can result in smoother activation and stable peak values.

[0046] During the maximum pooling operation 520, the training process 500c determines the smoothed maximum pooling loss, where the loss represents the difference between the output distribution that the network 300 theoretically believes to be and the actual output distribution. Here, the smoothed maximum pooling loss can be determined by the following equation: loss = loss + +loss - (1)

[0047]

number

[0048] Here, X t is a d-dimensional spectral feature, and y i (X t ,W) represents the i-th dimension of the softmax output of the neural network, W is the network weight, and C t is the frame label 420 in frame t (e.g., frame 210), and s(t) is the smoothing filter.

[0049]

number

[0050] This is a convolution over time,

[0051]

number

[0052] This defines the start and end times of the interval for the i-th maximum pooling window.

[0053] Continuing to refer to FIG. 5C, both the encoder 310c and the decoder 311c undergo a training process 500c that uses a smoothed max pooling technique. For example, FIG. 5C shows an encoder 310c that includes smoothing operations 510, 510e and max pooling operations 520, 520e. During the max pooling operation 520e of training 500c, the encoder 310c learns a sequence of sound portions (e.g., the vocal component of the audio feature 410) that define a hot word. Here, this learning can occur in a semi-supervised manner. In some examples, the max pooling operation 510e during training 500c divides a fixed-length hot word (e.g., a hot word of a desired length or an average length hot word) into max pooling windows 310w, 310w 1~n by splitting it.

[0054] For example, FIG. 6 shows n consecutive windows 310w over the expected hot word positions. The max pooling operation 510e then determines the max pooling loss in each window 310w. In some embodiments, the max pooling loss in each window 310w is defined by the following equation.

[0055]

Equation

[0056] Here, "e" corresponds to a variable of the encoder 310c, ω end corresponds to the end point of the hot word, and offset refers to the time offset of the window 310w.

[0057] In some examples, the number of windows 310w and / or the size 310w of each window 310w s are adjustable parameters during the training process 500c. These parameters are such that the number "n" of windows 310w approximates the number of distinguishable sound portions (e.g., phonemes), and / or the size 310w of the window 310w sThe number of windows 310w, "n", can be multiplied by this value to be adjusted to approximately match the fixed length of the hotword. (Number of windows 310w and size of each window 310w) s In addition to being adjustable, the endpoint of the hotword ω end Encoder offset to offset the sequence from window 310w, Offset e The variable called may also be adjustable during training 500c of encoder 310c.

[0058] Similar to encoder 310c, in training process 500c, decoder 311c includes smoothing operations 510, 510d, and max pooling operations 520, 520d. Generally speaking, training process 500c trains decoder 311c to produce strong activation (i.e., a high probability of detecting the hotword) for input frames 210 that contain audio features 410 at or near the end of the hotword. Due to the nature of max pooling loss, the max pooling loss value is such that decoder window 311w is at the actual endpoint ω of the hotword. end As long as it includes the end of the hot word ω end It is not affected by the exact value of . During the maximum pooling operation 520d of decoder 311c, the training process 500c determines the endpoint ω of the hotword according to the following formula. end Determine the maximum pooling loss of window 311w, which includes this window.

[0059]

number

[0060] Here, offset d and win size d The expected endpoint of the hot word is ω end It may be a parameter that can be adjusted to include [something].

[0061] Continuing to refer to Figure 6, the decoder window 311w is:

[0062]

number

[0063] from

[0064]

number

[0065] It is shown as an interval that extends to ω. The interval is the actual endpoint of the hot word. end When it is large enough to include, the smoothed maximum pooling loss technique allows network 300 to learn the optimal location of the strongest activation (e.g., semi-supervised). In some examples, the training process 500c learns the endpoint ω of the hot word based on word-level alignment. end We derive the following. In some embodiments, the endpoint ω of the hot word end This is determined based on the output of encoder 310.

[0066] In contrast to some end-to-end networks 300 that use co-training, in which encoder 310 is trained first and then decoder 311 can be trained while the model weights of encoder 310 are frozen, the smoothed maximum pooling method trains encoder 310c and decoder 311c simultaneously and co-train without such freezing. Since encoder 310c and decoder 311c are trained co-trained during the training process 500c using the smoothed maximum pooling loss, the relative importance of each loss can be controlled by an adjustable parameter α. For example, the total loss, referring to the loss in encoder 310c and the loss in decoder 311c, has a relationship expressed by the following equation: Total loss = α * loss e +loss d (10)

[0067] Referring here to Figure 7, the training process 700 for the memorized neural network 300 includes generating the first loss 710, 710a and the second loss 710, 710b, respectively, using the first label 420a (e.g., the cross-entropy label 420) and the second label 420b (e.g., the maximum pooling loss label), as well as the first loss function 705, 705a and the second loss function 705, 705b. Optionally, the process 700 begins by supplying the memorized neural network 300 with a training input audio sequence 400, where the data in the training input audio sequence 400 is labeled using both labels 420a and 420b. For example, a single training input audio sequence 400 is labeled using the first label 420a and the second label 420b, as previously described with respect to Figures 4A and 4B. The exemplary labels 420a and 420b are for illustrative purposes only and are not intended to limit any suitable labeling rules applicable to determining the loss 710 to be used in the training process 700.

[0068] Upon receiving the training input audio sequence 400, the memorized neural network 300 may generate an output 350 (i.e., a probability score 350). The memorized neural network 300 may process the training input audio sequence 400 in the manner described with respect to any of Figures 2 to 6, or in any other suitable manner for processing the audio data to determine the likelihood that a hotword is present in the training input audio sequence 400. In some embodiments, the output 350 is used by each of two loss functions 705. That is, the first loss function 705a receives the output 350 and label 420a to determine the first loss 710a. Similarly, the second loss function 705b receives the output 350 and label 420b to determine the second loss 710b. In particular, the loss 710 is determined from the same output 350 by using two different labels 420a and 420bb from the same training input audio sequence 400 and two different loss functions 705a and 705b, respectively. The loss function 705 can determine the loss 710 in any way as described with respect to any of Figures 2 to 6. In some examples, the first loss function 705a is a maximum pooling loss function and the second loss function 705b is a cross-entropy loss function. In other embodiments, a single loss function 705 receives the output 350 and labels 420 and generates a respective loss 710 based on each label 420. The loss function 705 can implement any suitable technique such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross-entropy, hinge loss, multiclass loss, etc.

[0069] In some embodiments, losses 710a and 710b are fed directly to the memorized neural network 300 during the training process 700. In other embodiments, losses 710a and 710b are combined or weighted together to produce combined losses 710 and 710c, which are processed by the memorized neural network 300. In some embodiments, the losses are averaged using a weighted average formula. For example, the first loss 710a and the second loss 710b may be defined as follows: First loss = L1[f(X,),Y1] (11) Second loss = L2[f(X,),Y2] (12)

[0070] Here, X is the output 350, L1 is the first loss function 705a, Y1 is the label 420a, L2 is the second loss function 705b, and Y2 is the label 420b. In these examples, the combined loss 710c is expressed as follows: Coupling loss=alpha*L1[f(X,theta),Y2]+beta*L2[f(X,theta),Y2] (13)

[0071] Here, alpha and beta are scalar hyperparameters. The first loss 710a and the second loss 710b may be combined in any other way (e.g., addition, multiplication, etc.).

[0072] The examples herein demonstrate training a neural network 300 using a training input audio sequence 400 annotated with two labels 420a and 420b. A first loss function 705a generates a first loss 710a using the output 350 and label 420a. A second loss function 705b generates a second loss 710b using the output 350 and label 420b. The neural network is trained, updated, or fine-tuned using both the first loss 710a and the second loss 710b. These examples are non-restrictive and should be understood that any number of labels 420 and any number of each loss function 705 can generate any number of losses to train any suitable neural network 300.

[0073] Figure 8 is a flowchart illustrating an exemplary configuration of operation for method 800, which trains a neural network 300 using multiple labels 420 and multiple loss functions 705. In operation 802, method 800 includes receiving a training input audio sequence 400, which includes a sequence of input frames, where the sequence of input frames defines a hotword that initiates the wake-up process on the user device 102. In operation 804, method 800 includes obtaining a first label 420a (e.g., a maximum pooling label) and a second label 420b (e.g., a cross-entropy label) for the training input audio sequence 400. In operation 806, method 800 includes using the stored neural network 300 and the training input audio sequence 400 to generate an output 350 indicating the likelihood that the training input audio sequence 400 contains the hotword. In operation 808, method 800 includes determining a first loss 710a (e.g., maximum pooling loss) based on a first label 420a and output 350. In operation 810, method 800 includes determining a second loss 710b (e.g., cross-entropy loss) based on a second label 420b and output 350. In operation 814, method 800 includes optimizing a stored neural network 300 based on the first loss 710a and the second loss 710b associated with the training input audio sequence 400.

[0074] As used herein, a software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be called an “application,” an “app,” or a “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0075] Non-temporary memory may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-temporary memory may also be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), phase-change memory (PCM), and disk or tape.

[0076] Figure 9 is a schematic diagram of an exemplary computing device 900 that may be used to carry out the systems and methods described herein. The computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are illustrative only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0077] The computing device 900 includes a processor 910, memory 920, storage device 930, a high-speed interface / controller 940 connected to memory 920 and a high-speed expansion port 950, and a low-speed bus 970 and a low-speed interface / controller 960 connected to storage device 930. Each component 910, 920, 930, 940, 950, and 960 is interconnected using various buses and may be mounted on a common motherboard or otherwise present as needed. The processor 910 processes instructions for execution within the computing device 900, including instructions stored in memory 920 or storage device 930, and may display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 970 connected to the high-speed interface 940. In other embodiments, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Also, multiple computing devices 900 may be connected so that each device provides a required portion of the operation (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0078] Memory 920 stores information non-temporarily within the computing device 900. Memory 920 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporarily stored memory 920 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 900. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disk or tape.

[0079] The storage device 930 can provide high-capacity storage to the computing device 900. In some embodiments, the storage device 930 is a computer-readable medium. In various different embodiments, the storage device 930 may be an array of devices including floppy disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid-state memory devices, or devices of a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that perform one or more of the above-described methods at runtime. The information carrier is a computer-readable medium or machine-readable medium such as memory 920, the storage device 920, or memory on the processor 910.

[0080] The high-speed controller 940 manages the bandwidth-intensive operation of the computing device 900, while the low-speed controller 960 manages the less bandwidth-intensive operation. This assignment of roles is merely an example. In some embodiments, the high-speed controller 940 is coupled to memory 920, a display 980 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 950 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 960 is coupled to a storage device 930 and a low-speed expansion port 990. The low-speed expansion port 990 may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet) and may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or network devices such as switches or routers, for example, via a network adapter.

[0081] The computing device 900 can be implemented in many different forms, as shown in the figure. For example, it may be implemented as a standard server 900a, or multiple times within a group of such servers 900a, as a laptop computer 900b, or as part of a rack server system 900c.

[0082] Various embodiments of the systems and technologies described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be specific or general-purpose and may include implementations in one or more computer programs that are executable and / or interpretable in a programmable system comprising at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to a storage system.

[0083] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” mean any computer program product, non-transient computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor that includes a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signals” means any signals used to provide machine instructions and / or data to a programmable processor.

[0084] The processes and logic flows described herein, also known as data processing hardware, can be executed by one or more programmable processors that perform functions by executing one or more computer programs, operating on input data, and producing outputs. Processors and logic flows can also be executed by dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Processors suitable for executing computer programs include, by example, both general-purpose and dedicated microprocessors, as well as any one or more processors in any type of digital computer. Generally, processors receive instructions and data from read-only memory, random-access memory, or both. The basic elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely coupled to receive data from them, transfer data to them, or both. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices like EPROM, EEPROM, and flash memory devices; magnetic disks like internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be complemented by or incorporated into dedicated logic circuits.

[0085] To provide user interaction, one or more aspects of this disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor) or a touchscreen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) on which the user can provide input to the computer. Other types of devices may also be used to provide user interaction, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and the input from the user may be received in any form, including acoustic, spoken language, or haptic input. Furthermore, the computer may interact with the user by sending and receiving documents to and from the user's device, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0086] Several embodiments are described. Needless to say, it is understood that various modifications can be made without departing from the spirit and scope of this disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A method (800) performed by a computer that causes the data processing hardware (103, 112) to perform an operation, when the operation is performed by the data processing hardware (103, 112), wherein the operation is Receiving a training input audio sequence (400) including a sequence of input frames, wherein the sequence of input frames defines a hotword that initiates a wake-up process on a user device (102), Obtaining a first label (420, 420a) for the aforementioned training input audio sequence, Obtaining a second label (420, 420b) for the training input audio sequence, wherein the second label (420, 420b) is different from the first label (420, 420a), Using the stored neural network (300) and the training input audio sequence (400), an output (350) is generated that indicates the likelihood that the training input audio sequence (400) contains the hot word. The first loss (710, 710a) is determined based on the first label (420, 420a) and the output (350), Determining a second loss (710, 710b) based on the second labels (420, 420b) and the output (350), The stored neural network (300) is optimized based on the first loss (710, 710a) and the second loss (710, 710b) associated with the training input audio sequence (400), Includes, The first label (420, 420a) includes a maximum pooling loss label, and in the maximum pooling loss label, A target label with the number "1" is assigned to the sequence of input frames corresponding to the last audio component of the hot word. A target label with the number "0" is assigned to the sequence of input frames corresponding to the audio components of the hot word, excluding the last audio component. The second label (420, 420b) includes a cross-entropy label, the cross-entropy label is associated with a score that increases along the sequence of input frames as the number of audio features characterizing the speech components of the hot word increases, The first loss (710, 710a) includes the maximum pooling loss, the maximum pooling loss represents the error between the output (350) and the maximum pooling loss label. The second loss (710, 710b) includes the cross-entropy loss, which represents the error between the output (350) and the cross-entropy label. The aforementioned method (800).

2. The method according to claim 1 (800), wherein the stored neural network (300) includes an encoder (310) and a decoder (311), and each of the encoder (310) and the decoder (311) of the stored neural network (300) includes a sequentially stacked single value decomposition filter (SVDF) layer (302).

3. The method (800) according to claim 1 or 2, wherein the output (350) is based on a probability distribution of logits (502) based on the training input audio sequence (400).

4. The operation further includes smoothing the logit (502) based on the training input audio sequence (400) before determining the first loss (710, 710a), The method according to claim 1 or 2 (800).

5. Determining the first loss (710, 710a) is The process involves generating a plurality of encoder windows (310w), wherein each encoder window of the plurality of encoder windows (310w) is associated with one or more phonemes of the hotword. The first loss (710, 710a) is determined for each encoder window of the plurality of encoder windows (310w), The method according to claim 1 or 2 (800), including the method according to claim 1 or 2.

6. The method according to claim 5 (800), wherein the combined size of the plurality of encoder windows (310w) corresponds to the average acoustic length of the hot word.

7. The method according to claim 1 or 2 (800), wherein optimizing the stored neural network (300) includes generating a weighted average of the first loss (710, 710a) and the second loss (710, 710b).

8. The method according to claim 1 or 2 (800), wherein the second label (420, 420b) is derived from one or more phoneme sequences of the hotword.

9. The method according to claim 1 or 2 (800), wherein the first label (420, 420a) is based on the position of the last phoneme of the hot word.

10. The method according to claim 1 or 2 (800), wherein each sequence of the input frames includes one or more respective audio features (410) that characterize the audio component (430) of the hotword.

11. System (100), Data processing hardware (103, 112) and The system includes memory hardware (105, 114) that communicates with the data processing hardware (103, 112), and when the memory hardware (105, 114) is executed on the data processing hardware (103, 112), it stores instructions that cause the data processing hardware (103, 112) to perform an operation, and the operation is Receiving a training input audio sequence (400) including a sequence of input frames, wherein the sequence of input frames defines a hotword that initiates a wake-up process on a user device (102), Obtaining a first label (420, 420a) for the aforementioned training input audio sequence, Obtaining a second label (420, 420b) for the training input audio sequence, wherein the second label (420, 420b) is different from the first label (420, 420a), Using the stored neural network (300) and the training input audio sequence (400), an output (350) is generated that indicates the likelihood that the training input audio sequence (400) contains the hot word. The first loss (710, 710a) is determined based on the first label (420, 420a) and the output (350), Determining a second loss (710, 710b) based on the second labels (420, 420b) and the output (350), Optimizing the stored neural network (300) based on the first loss (710, 710a) and the second loss (710, 710b) associated with the training input audio sequence, Includes, The first label (420, 420a) includes a maximum pooling loss label, and in the maximum pooling loss label, A target label with the number "1" is assigned to the sequence of input frames corresponding to the last audio component of the hot word. A target label with the number "0" is assigned to the sequence of input frames corresponding to the audio components of the hot word, excluding the last audio component. The second label (420, 420b) includes a cross-entropy label, the cross-entropy label is associated with a score that increases along the sequence of input frames as the number of audio features characterizing the speech components of the hot word increases, The first loss (710, 710a) includes the maximum pooling loss, the maximum pooling loss represents the error between the output (350) and the maximum pooling loss label. The second loss (710, 710b) includes the cross-entropy loss, which represents the error between the output (350) and the cross-entropy label. The aforementioned system (100).

12. The system (100) according to claim 11, wherein the stored neural network (300) includes an encoder (310) and a decoder (311), and each of the encoder (310) and the decoder (311) of the stored neural network (300) includes a sequentially stacked single value decomposition filter (SVDF) layer (302).

13. The system (100) according to claim 11 or 12, wherein the output (350) is based on a probability distribution of logits (502) based on the training input audio sequence (400).

14. The system (100) according to claim 11 or 12, further comprising smoothing a logit (502) based on the training input audio sequence (400) before determining the first loss (710, 710a).

15. Determining the first loss (710, 710a) is The process involves generating a plurality of encoder windows (310w), wherein each encoder window of the plurality of encoder windows (310w) is associated with one or more phonemes of the hotword. The first loss (710, 710a) is determined for each encoder window of the plurality of encoder windows (310w), The system (100) according to claim 11 or 12, including the system described in claim 11 or 12.

16. The system (100) according to claim 15, wherein the combined size of the plurality of encoder windows (310w) corresponds to the average acoustic length of the hot word.

17. The system (100) according to claim 11 or 12, wherein optimizing the stored neural network (300) includes generating a weighted average of the first loss (710, 710a) and the second loss (710, 710b).

18. The system (100) according to claim 11 or 12, wherein the second label (420, 420b) is derived from one or more phoneme sequences of the hotword.

19. The system (100) according to claim 11 or 12, wherein the first label (420, 420a) is based on the position of the last phoneme of the hot word.

20. The system (100) according to claim 11 or 12, wherein each sequence of the input frames includes one or more respective audio features (410) that characterize the audio component (430) of the hotword.

Images