Optimizing Personal VAD for On-Device Speech Recognition

Abstract

The computer-implemented method (500) includes receiving a sequence of acoustic frames (110) corresponding to an utterance and generating a reference speaker embedding (342) for the utterance. The method also includes receiving a target speaker embedding (412) for the target speaker and generating feature-by-feature linear modulation (FiLM) parameters (335) including a scaling vector (332) and a shift vector (334) based on the target speaker embedding. The method also includes generating an affine transformation output (352) that scales and shifts the reference speaker embedding based on the FiLM parameters. The method also includes generating a classification output (362) indicating whether the utterance was spoken by the target speaker based on the affine transformation output.

Description

[Technical field]

[0001] This disclosure relates to optimizing a personal voice activity detector for on-device voice recognition. [Background technology]

[0002] Voice-enabled devices have grown in popularity over the past few years. One challenge for voice-enabled devices is the ability to distinguish between background noise from the surrounding environment and voices directed to the device. In some examples, the voice-enabled device further determines whether the voices directed to the device are spoken by a particular user or by other users. This ability allows the device to decide whether to further process the audio (e.g., process a command or query) or simply ignore the received audio. Given the latency and computational constraints of certain voice-enabled devices in real-world environments, the ability of the device to distinguish between background noise and voices spoken by a particular user becomes even more challenging. Summary of the Invention

[0003] One aspect of the present disclosure provides a personalized voice activity detector (VAD). The personalized VAD includes a stack of multi-headed self-attention blocks configured to receive as an input a sequence of acoustic frames corresponding to an utterance and generate as an output a reference speaker embedding for the utterance. The personalized VAD also includes a feature-wise linear modulation (FiLM) generator configured to receive as an input a target speaker embedding of a target speaker and generate as an output FiLM parameters including a scaling vector and a shift vector based on the target speaker embedding. The personalized VAD also includes a FiLM layer configured to receive as input the reference speaker embedding and the FiLM parameters and generate as an output an affine transformation output that scales and shifts the reference speaker embedding based on the FiLM parameters. The personalized VAD also includes a classifier configured to generate a classification output indicative of whether the utterance was spoken by the target speaker based on the affine transformation output.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the classification includes at least one of a target speaker token, a non-target speaker token, or a non-speech token. In some examples, the personal VAD further includes a speaker pre-net configured to receive as an input a sequence of acoustic frames and generate as an output a speaker information embedding extracted from the sequence of acoustic frames. In these examples, the FiLM generator may be further configured to receive as an input a cosine similarity between the target speaker embedding and the target information embedding and generate as an output FiLM parameters including a scaling vector and a shift vector based on the cosine similarity. Here, the speaker pre-net includes a stack of multi-head self-attention layers including one or more conformer layers.

[0005] The stack of multi-head self-attention blocks may include one or more conformer layers. In some implementations, the classifier includes a fully connected layer. The personal VAD may operate in a streaming manner. In some examples, the personal VAD further includes a pre-trained text-independent speaker recognition model configured to receive as input an enrollment utterance spoken by a target speaker and generate as output a target speaker embedding for the target speaker based on the enrollment utterance. In some implementations, the personal VAD is trained with training data including enrollment training utterances paired with a target speaker embedding and non-enrollment training utterances not paired with any corresponding target speaker embedding.

[0006] Another aspect of the disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for optimizing personal voice activity detection. The operations include receiving a sequence of acoustic frames corresponding to an utterance as an input to a personal voice activity detector (VAD). The operations also include generating a reference speaker embedding for the utterance using a stack of multi-head self-attention blocks of the personal VAD. The operations also include receiving a target speaker embedding of the target speaker as an input to a feature-wise linear modulation (FiLM) generator of the personal VAD, and generating FiLM parameters including a scaling vector and a shift vector based on the target speaker embedding using the FiLM generator. The operations also include generating an affine transformation output using a FiLM layer of the personal VAD that scales and shifts the reference speaker embedding based on the FiLM parameters. The operations also include generating a classification output using a classifier of the personal VAD based on the affine transformation output, indicating whether the utterance was spoken by the target speaker.

[0007] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the classification output includes at least one of a target speaker token, a non-target speaker token, or a non-speech token. In some examples, the operations further include using a speaker pre-net of the personal VAD to generate a speaker information embedding extracted from the sequence of acoustic frames. In these examples, the operations may further include using a FiLM generator to generate FiLM parameters based on a cosine similarity between the target speaker embedding and the speaker information embedding. Here, the speaker pre-net includes a stack of multi-head self-attention layers including one or more conformer layers.

[0008] The stack of multi-head self-attention blocks may include one or more conformer layers. In some implementations, the classifier includes a fully connected layer. The personal VAD may operate in a streaming manner. In some examples, the operation further includes receiving an enrollment utterance spoken by a target speaker as an input to a pre-trained text-independent speaker recognition model, and generating a target speaker embedding of the target speaker based on the enrollment utterance using the pre-trained text-independent speaker recognition model. The personal VAD may be trained with training data including enrollment training utterances paired with the target speaker embeddings and non-enrollment training utterances not paired with any corresponding target speaker embeddings.

[0009] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram of an example speech recognition system including a speech recognition model. [Diagram 2]FIG. 1 is a schematic diagram of an example speech recognition model including a recurrent neural network transducer (RNN-T) model architecture. [Figure 3A] 1 is a schematic diagram of an exemplary personal voice activity detector; [Figure 3B] 1 is a schematic diagram of an exemplary personal voice activity detector; [Figure 4] FIG. 2 is a schematic diagram of the enrollment process of a target speaker. [Diagram 5] 3B is a flowchart of an example arrangement of operations for a method of optimizing a personal voice activity detector using the example personal voice activity detector of FIG. 3A. [Figure 6] 3C is a flowchart of an example arrangement of operations for a method of optimizing a personal voice activity detector using the example personal voice activity detector of FIG. 3B. [Figure 7] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0011] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Many speech recognition systems include a voice activity detector (VAD) that resides upstream of other components of the speech recognition system, such as an automatic speech recognition (ASR) model and a speaker identification model. Here, the VAD acts as a gating component that discards acoustic frames that contain non-speech data (e.g., television noise or other background noise) and forwards acoustic frames that contain speech data to downstream components of the speech recognition system. Thus, the VAD improves the performance of downstream components (e.g., the ASR model) by preventing more computationally expensive downstream components from processing acoustic frames that do not contain any non-speech data, thereby reducing the overall computational cost and size of the speech recognition system.

[0013] Recently, VADs have been personalized for a target user (or multiple target users) such that the personalized VAD discards any acoustic frames that do not contain speech spoken by the target user. That is, while a conventional VAD only determines whether the input audio contains non-speech or speech (e.g., speech spoken by any user), the personalized VAD determines whether the input audio contains non-speech, speech spoken by the target user (or one of multiple target users), or speech spoken by a non-target user. Here, downstream components of the speech recognition system only process the acoustic frames spoken by the target user.

[0014] However, current personalized VADs have several significant drawbacks that prevent them from being used in production speech recognition systems. For example, current personalized VADs determine whether the input audio data contains non-speech, speech by a target user, or speech by a non-target user by concatenating the input audio data with speaker embeddings. Importantly, acoustic features and speaker embeddings represent very different information and are extracted through completely separate processes, and therefore have different distributions and magnitudes. Therefore, simply concatenating the input audio data with speaker embeddings would significantly limit the capabilities of these personalized VADs. As a result of this concatenation approach, the word error rate (WER) of the speech recognition system is degraded, and as a result, personalized VADs are not suitable for production speech recognition systems.

[0015] Another significant shortcoming of current personalized VAD is that it assumes that at least one target speaker is enrolled. Enrolling a target speaker involves prompting the target speaker to speak enrollment utterances, encoding the target user's voice characteristics, and generating a speaker embedding (i.e., enrollment scenario). However, the enrollment process is optional, and in many cases, users skip the enrollment process, resulting in zero enrolled target speakers for a particular device (i.e., no enrollment scenario). As a result, assuming the presence of at least one target speaker during personalized VAD training can have a detrimental effect on the speech recognition system in the absence of enrolled target speakers in a production environment.

[0016] Accordingly, embodiments herein are directed to a personalized VAD optimized for speech recognition. In some embodiments, the personalized VAD includes a stack of multi-head self-attention blocks, a feature-by-feature linear modulation (FiLM) generator, a FiLM layer, and a classifier. The stack of multi-head self-attention blocks is configured to generate a reference speaker embedding for the utterance, and the FiLM generator is configured to generate FiLM parameters based on a target speaker embedding of a target speaker. The FiLM layer is then configured to generate an affine transformation that scales and shifts the reference speaker embedding based on the FiLM parameters, and the classifier is configured to determine whether the utterance is spoken by the target speaker based on the affine transformation output.

[0017] In another embodiment, the personal VAD also includes a speaker pre-net configured to generate a speaker information embedding extracted from the utterance and determine a cosine similarity between the target speaker embedding and the speaker information embedding. Here, the FiLM generator generates the FiLM parameters based on the cosine similarity (rather than directly based on the target speaker embedding). The personal VAD discards any acoustic frames that are not spoken by the target speaker and sends the acoustic frames that are spoken by the target speaker to downstream components for further processing. In particular, the personal VAD operates in a streaming manner by generating, for each acoustic frame in a sequence of acoustic frames of the utterance, a frame-by-frame decision indicating whether the corresponding acoustic frame was spoken by the target speaker. As will become apparent, the personal VAD is trained using training data that includes training utterances paired with the target speaker embedding and training utterances that are not paired with any target speaker embedding. Training the personal VAD in this manner allows the speech recognition system to maintain optimal performance in terms of WER and latency in both enrollment and no-enrollment scenarios.

[0018] 1 illustrates an automatic speech recognition (ASR) system 100. The automatic speech recognition (ASR) system 100 implements a neural network model (e.g., ASR model) 200 and a personal voice activity detector (VAD) 300, each of which resides on a user device 102 of a user 104 and / or a remote computing device 60 (e.g., one or more servers of a distributed system running in a cloud computing environment) that communicates with the user device 102. Although the user device 102 is illustrated as a mobile computing device (e.g., a smartphone), the user device 102 may correspond to any type of computing device, such as, without limitation, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart home appliance, an automobile infotainment system, or an Internet of Things (IoT) device, and includes data processing hardware 111 and memory hardware 113.

[0019] The user device 102 includes an audio subsystem 108 configured to receive utterances 106 spoken by a user 104 (e.g., the user device 102 may include one or more microphones for recording the spoken utterance 106) and convert the utterance 106 into a corresponding digital format associated with a sequence of acoustic frames 110 that can be processed by the ASR system 100. In the illustrated example, the user speaks the phrase “What's the weather like in New York City?” in the natural language of English as the utterance 106, and the audio subsystem 108 converts the utterance 106 into a corresponding sequence of acoustic frames 110 for input to the ASR system 100.

[0020] The personal VAD 300 receives as input a sequence of acoustic frames 110 corresponding to an utterance 106 and generates a classification output indicating whether the utterance 106 was spoken by a target speaker. In some examples, the personal VAD 300 discards acoustic frames 110 that contain non-speech or speech not spoken by a target speaker. Here, the personal VAD 300 sends only acoustic frames 110 that contain speech spoken by a target speaker to the ASR model 200 for further processing (e.g., in an enrollment scenario with at least one target speaker). In other examples, the personal VAD 300 discards acoustic frames 110 only when the acoustic frames 110 contain non-speech (e.g., in a no-enrollment scenario with zero target speakers) and sends acoustic frames 110 that contain speech spoken by any user to the ASR model 200 for further processing. In particular, the personal VAD 300 may enable the target speaker to speak to the ASR system 100 without speaking a wake-up phrase (e.g., hot word / wake word) to wake up the ASR model 200 and begin processing the input audio data to transcribe the speech. In some examples, the user device 102 operates in a low power state, and when the personal VAD 300 classifies the input acoustic frames 110 as speech spoken by the target speaker, it causes the user device 102 to wake up from the low power state and invoke the speech recognition model 200 to perform speech recognition on the input acoustic frames 110.

[0021] The ASR model 200 then receives as input a sequence of acoustic frames 110 corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 (e.g., recognition result / hypothesis) of the utterance 106. In the illustrated example, the user device 102 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120 (e.g., recognition result / hypothesis) of the utterance 106. In the illustrated example, the user device 102 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to a user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 100 is processed by a natural language understanding (NLU) module executing on the remote computing device 60 of the user device 102, for example, to execute user commands. Additionally or alternatively, a text-to-speech model (e.g., running on any combination of the user device 102 or the remote computing device 60) may convert the transcription 120 into synthetic speech for audible output by another device. For example, the original utterance 106 may be sent to a friend, and the transcription 120 may be converted to synthetic speech and audibly output so that the friend can hear the message conveyed in the original utterance 106.

[0022] Referring to FIG. 2, an exemplary ASR model 200 may include a recurrent neural network-transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The use of the RNN-T model architecture is merely exemplary, and the ASR model 200 may include other architectures, such as a transformer-transducer model architecture and a conformer-transducer model architecture, among others. Because the RNN-T model 200 uses fewer computational resources and has lower memory requirements than traditional ASR architectures, the RNN-T model architecture is suitable for performing speech recognition entirely on the user device 102 (e.g., without requiring communication with a remote server). The RNN-T model 200 includes an encoder network 210, a predictive network 220, and a combination network 230. The encoder network 210, which is roughly similar to an acoustic model (AM) in traditional ASR systems, includes a recurrent network of stacked very short-term memory (LSTM) layers. For example, the encoder network (e.g., encoder) 210 may include a scalar network of x t ∈R d Given a sequence of d-dimensional feature vectors (e.g., a sequence of acoustic frames 110 (FIG. 1)) x=(x1, x2, . . . , x T ), and the encoder network 210 generates a high-level feature representation 212 at each of a number of output steps.

number

[0023] Similarly, the prediction network 220 is also an LSTM network, and like the language model (LM), it is calculated based on the sequence of non-empty symbols y0,...,y ui-1 A dense representation of

number

number

[0024] The softmax layer 240 may use any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this way, the RNN-T model 200 does not make any conditional independent assumptions, but rather the prediction of each symbol depends not only on the acoustic frame but also on the sequence of labels output so far. Thus, the softmax layer 240 may select the speech recognition hypothesis with the highest corresponding probability as the transcription 120 from the probability distribution. The RNN-T model 200 assumes that the output symbols are independent of future acoustic frames 110, which allows the RNN-T model 200 to be used in a streaming manner.

[0025] In some examples, the encoder 210 of the RNN-T model includes multiple multi-head (e.g., 8-head) self-attention layers. For example, the multiple multi-head self-attention layers may include a conformer layer (e.g., a conformer-encoder), a transformer layer, a performer layer, a convolutional layer (including a lightweight convolutional layer), or any other type of multi-head self-attention layer. The multiple multi-head self-attention layers may include any number of layers, for example, 16 layers. Furthermore, the encoder 210 may operate in a streaming manner (e.g., the encoder 210 outputs the higher-level feature representations 212 immediately as they are generated) or in a non-streaming manner in which the encoder 210 processes additional right context to receive and improve the speech recognition results.

[0026] 3A and 3B show an exemplary personal VAD 300. The personal VAD 300 functions as a gating component of the ASR system 100, discarding the acoustic frames 110 that do not include speech uttered by the target speaker. On the other hand, the personal VAD sends the acoustic frames 110 that include speech uttered by the target speaker to the ASR model 200 (FIG. 2) for further processing. Notably, in some scenarios, the personal VAD 300 operates as a non-personalized VAD and sends the acoustic frames 110 that include speech uttered by any user (e.g., regardless of whether it is uttered by the target speaker or by other speakers) to the ASR model 200 (FIG. 2) for further processing. For example, even in a no-enrollment scenario (e.g., when the target speaker is not enrolled), the personal VAD 300 may send the acoustic frames 110 that include speech for further processing and discard the acoustic frames 110 that include non-speech.

[0027] FIG. 3A illustrates an exemplary personal VAD 300, 300a including a FiLM generator 330, a stack of multi-head self-attention blocks 340, a FiLM layer 350, and a classifier 360. The stack of multi-head self-attention blocks 340 (also simply referred to as a "stack of self-attention blocks 340") includes one or more conformer layers (e.g., four conformer layers), where each conformer layer includes a 64-dimensional conformer layer, a multi-head (e.g., eight-head) attention mechanism, a 7×7 convolution kernel showing causality, and a 31-frame left context. In another example, each self-attention block 340 in the stack of self-attention blocks 340 includes one or more other self-attention layers, such as, for example, a transformer layer, a performer layer, or a convolutional layer.

[0028] The stack of self-attention blocks 340 is configured to receive as input a sequence of acoustic frames (x) 110 corresponding to an utterance, and generate, at each of a number of output steps, a reference speaker embedding (h) 342 for a corresponding acoustic frame 110 in the sequence of acoustic frames 110 of the utterance. For example, when the stack of self-attention blocks includes one or more conformer layers, the reference speaker embedding is represented as h=conformer(x). The stack of self-attention blocks 340 outputs the generated reference speaker embedding at each output step to the FiLM layer 350.

[0029] The FiLM generator 330 takes as input the target speaker embedding (e target ) 412 and generates as output the FiLM parameters 335 based on the target speaker embeddings 412, where the FiLM parameters 335 include a scaling vector (γ(e target ))332 and the shift vector (β(e target ) 334 and (collectively referred to as FiLM parameters 335). As will become apparent, the FiLM layer 350 uses the FiLM parameters 335 to modulate the reference speaker embeddings 342. In other words, the FiLM generator 330 generates the FiLM parameters 335 based on external conditioning inputs (e.g., the target speaker embeddings 412). The FiLM generator 330 outputs the scaling vector 332 and the shift vector 334 to the FiLM layer 350.

[0030] The FiLM layer is configured to receive as input the reference speaker embeddings 342 generated by the stack of self-attention blocks 340 and the FiLM parameters 335 generated by the FiLM generator 330 at each of the multiple output steps, and generate the affine transformed output 352 at each of the multiple output steps. The FiLM layer 350 generates the affine transformed output 352 by applying a feature-wise affine transformation (e.g., a FiLM operation) to the reference speaker embeddings 342 using the FiLM parameters 335 (e.g., the scaling vector 332 and the shift vector 334). Notably, the feature-wise affine transformation generalizes the concatenation-based, bias-based, and scaling-based conditioning operators, which are more expressive than using any one individually in learning the conditioning representations.

[0031] In some implementations, the FiLM layer 350 applies a different affine transform to each feature of the reference speaker embedding 342. In other implementations, the FiLM layer 350 applies a different affine transform to each channel that is consistent across spatial locations (e.g., in a convolutional network configuration). For example, in these implementations, the FiLM layer 350 first applies a scaling vector (γ(e target ) 332 to scale each feature (or channel) of the reference speaker embedding 342 and then scale the shift vector (β(e target )) 334 is used to shift each feature (or channel) of the reference speaker embedding 342. In particular, the FiLM layer 350 may generate an affine transformed output 352 according to: FiLM(h)=(γ(e target ))*h+(β(e target ))(1) In Equation 1, FiLM(h) represents the affine transformation output 352, and γ(e target ) represents the scaling vector 332, and β(e target ) represents the shift vector 334 and h represents the reference speaker embedding 342.

[0032] In some implementations, the classifier 360 includes a fully connected layer. The classifier 360 is configured to receive as input the affine transform output 352 generated by the FiLM layer 350 at each of the multiple output steps and generate a classification output 362 indicating whether the utterance was spoken by the target speaker based on the affine transform output 352. The classification output 362 may include at least one of a target speaker token (tst) indicating that the utterance was spoken by the target speaker, a non-target speaker token (ntst) indicating that the utterance was spoken by another speaker (e.g., a non-target speaker), or a non-speech token (nst) indicating that the utterance was non-speech. For example, the non-speech may include audio data representing audio data captured from a television or radio.

[0033] 3B shows an exemplary personal VAD 300, 300b including a speaker pre-net 310, a comparator 320, a FiLM generator 330, a stack of self-attention blocks 340, a FiLM layer 350, and a classifier 360. That is, instead of directly conditioning the target speaker embedding 412 to the FiLM generator 330, the speaker pre-net 310 extracts speaker information embeddings 312 from the acoustic frame 110. The speaker information embeddings 312 include the same dimensions as the target speaker embeddings 412. As an advantage, using the speaker information embeddings 312 provides more characteristic information to the personal VAD 300, which allows the classifier 360 to more accurately determine whether the acoustic frame 110 was spoken by the target speaker.

[0034] In particular, the speaker pre-net 310 is configured to receive a sequence of acoustic frames (x) 110 and generate, at each of a plurality of output steps, a speaker information embedding 312 for a corresponding acoustic frame 110 in the sequence of acoustic frames 110. That is, the speaker pre-net 310 extracts the speaker information embedding 312 from the sequence of acoustic frames 110. The speaker pre-net 310 may generate the speaker information embedding 312 according to the following: e prenet = PreNet(x)(2) In Equation 2, e prenet represents a speaker information embedding 312, and x represents a sequence of acoustic frames 110. Each speaker information embedding 312 includes a fixed-length embedding of the corresponding acoustic frame 110, represented by:

number

[0035] The comparator 320 receives as input the speaker information embeddings 312 and the target speaker embeddings 412 generated by the speaker pre-net 310 at each of the multiple output steps, and generates as output a cosine similarity score 322. In particular, the comparator 320 calculates the cosine similarity score (s∈R T ) 322. The comparator 320 may generate a cosine similarity score, represented by: s = cos(e prenet ,e target )(4)

[0036] The FiLM generator 330 is configured to receive as an input the cosine similarity score(s) 322 and generate as an output the FiLM parameters 335 based on the cosine similarity score 322, where the FiLM parameters 335 include a scaling vector γ(s) 332 and a shift vector β(s) 334 (collectively referred to as FiLM parameters 335). The FiLM layer 350 uses the FiLM parameters 335 to modulate the reference speaker embedding 342. The FiLM generator 330 outputs the scaling vector 332 and the shift vector 334 to the FiLM layer 350.

[0037] The stack of self-attention blocks 340 is configured to receive as input a sequence of acoustic frames (x) 110 corresponding to an utterance, and generate, at each of a number of output steps, a reference speaker embedding (h) 342 for a corresponding acoustic frame 110 in the sequence of acoustic frames 110 of the utterance. For example, when the stack of self-attention blocks includes one or more conformer layers, the reference speaker embedding is represented as h=conformer(x). The stack of self-attention blocks 340 outputs the generated reference speaker embedding at each output step to the FiLM layer 350.

[0038] The FiLM layer is configured to receive as input the reference speaker embeddings 342 generated by the stack of self-attention blocks 340 and the FiLM parameters 335 generated by the FiLM generator 330 at each of the multiple output steps, and to generate the affine transformed outputs 352 at each of the multiple output steps. The FiLM layer 350 generates the affine transformed outputs 352 by applying a feature-by-feature affine transformation (e.g., a FiLM operation) to the reference speaker embeddings 342 using the FiLM parameters 335 (e.g., the scaling vectors 332 and the shift vectors 334).

[0039] In some implementations, the FiLM layer 350 applies a different affine transform to each feature of the reference speaker embedding 342. In other implementations, the FiLM layer 350 applies a different affine transform to each channel that is consistent across spatial locations (e.g., in a convolutional network configuration). For example, in these implementations, the FiLM layer 350 first applies a scaling vector (γ(e target ) 332 to scale each feature (or channel) of the reference speaker embedding 342 and then scale the shift vector (β(e target )) 334 is used to shift each feature (or channel) of the reference speaker embedding 342. In particular, the FiLM layer 350 may generate an affine transformed output 352 according to: FiLM(h) = γ(s) * x + β(s) (5) In Equation 1, FiLM(h) represents the affine transformation output 352, and (γ(e target )) represents the scaling vector 332, and (β(e target )) represents the shift vector 334 and h represents the reference speaker embedding 342.

[0040] In some implementations, the classifier 360 includes a fully connected layer. The classifier 360 is configured to receive as input the affine transform output 352 generated by the FiLM layer 350 at each of the multiple output steps and generate a classification output 362 indicating whether the utterance was spoken by the target speaker based on the affine transform output 352. The classification output 362 may include at least one of a target speaker token (tss) indicating that the utterance was spoken by the target speaker, a non-target speaker token (ntss) indicating that the utterance was spoken by a speaker different from the target speaker (e.g., a non-target speaker), or a non-speech token (ns) indicating that the utterance was non-speech. For example, the non-speech may include audio data representing audio data captured from a television or radio.

[0041] 3A and 3B, in an enrollment scenario, the personal VAD 300 may transmit the acoustic frames 110 to the ASR model 200 (FIG. 2) for further processing only when the personal VAD 300 generates a classification output 362 indicative of a target speaker token for the corresponding acoustic frame 110, and discard the acoustic frames 110 corresponding to non-target speaker tokens or non-speech tokens. Meanwhile, in a no-enrollment scenario, the exemplary personal VAD 300a may transmit the acoustic frames 110 to the ASR model 200 (FIG. 2) for further processing only when the personal VAD 300 generates a classification output 362 indicative of either a target speaker token or a non-target speaker token for the corresponding acoustic frame 110, and discard the acoustic frames 110 corresponding to non-speech talk. Notably, the personal VAD 300 operates in a streaming manner such that the personal VAD outputs a classification output 362 for a corresponding acoustic frame 110 in a sequence of acoustic frames 110 at each of a plurality of output steps. Therefore, the Personal VAD 300 can be seamlessly integrated into ASR systems, including streaming ASR models, with minimal (or no) additional latency.

[0042] 4 illustrates a target speaker enrollment process 400 configured to generate a target speaker embedding 412 for a target speaker. That is, to determine whether an utterance is spoken by a target speaker, a personal VAD requires a target speaker embedding 412 that represents the speech characteristics of the target speaker. Thus, a user device 102 (FIG. 1) may prompt a user 104 to speak one or more enrollment utterances 402 to encode the speech characteristics of the user 104. The target speaker enrollment process 400 includes a pre-trained text-independent speaker recognition model 410 (sometimes simply referred to as “speaker recognition model 410”) that receives the enrollment utterances 402 and generates a target speaker embedding 412 for the corresponding user 104.

[0043] The target speaker enrollment process 400 may pair the generated target speaker embedding 412 with a user to make the user a target speaker. The target speaker embedding 412 may be stored in the memory hardware 113 of the user device 102 (FIG. 1). While the above-described implementation considers only a single target speaker, the target speaker enrollment process 400 may generate target speaker embeddings 412 corresponding to each target speaker of multiple target speakers. For example, each target speaker may correspond to a respective enrolled user among multiple enrolled users who share a voice-enabled device.

[0044] In some examples, the user declines the target speaker enrollment process 400, resulting in zero target speakers. Thus, the personal VAD 300 is trained using training data that includes utterances spoken by target speakers with corresponding target speaker embeddings 412 and utterances spoken by non-target speakers (e.g., non-enrollment speakers) without corresponding target speaker embeddings 412. In particular, the training process generates training data by sampling a subset of training utterances that include the target speaker embeddings 412 and ground truth labels of non-target speaker tokens, setting the target speaker embeddings to zero, and replacing the non-target speaker tokens with the target speaker tokens. Advantageously, by generating training data in this manner, the speech recognition system can maintain performance (e.g., WER) in enrollment and non-enrollment scenarios.

[0045] 5 is a flow chart of an exemplary arrangement of operations of a method 500 for optimizing a personal voice activity detector. The method 500 may be executed in data processing hardware 710 (FIG. 7) using instructions stored in memory hardware 720 (FIG. 7). The data processing hardware 710 and memory hardware 720 may be present in the user device 102 and / or the remote computing device 60 of FIG. 1, which corresponds to the computing device 700 (FIG. 7).

[0046] At operation 502, the method 500 includes receiving a sequence of acoustic frames 110 corresponding to an utterance as an input to the personal VAD 300. At operation 504, the method 500 includes generating a reference speaker embedding 342 of the utterance using a stack of multi-head self-attention blocks 340 of the personal VAD. At operation 506, the method 500 includes receiving a target speaker embedding 412 of the target speaker as an input to a FiLM generator 330, and at operation 508, using the FiLM generator 330 to generate FiLM parameters 335 including a scaling vector 332 and a shift vector 334 based on the target speaker embedding 412. At operation 510, the method 500 includes using a FiLM layer 350 to generate an affine transform output 352 that scales and shifts the reference speaker embedding 342 based on the FiLM parameters 335. At operation 512, the method 500 includes generating a classification output 362 based on the affine transformation output 352, the classification output 362 indicating whether the utterance (or the corresponding acoustic frame 110) was spoken by the target speaker.

[0047] 6 is a flowchart of an exemplary arrangement of operations for a method 600 of optimizing a personal voice activity detector. The method 600 may be executed in data processing hardware 710 (FIG. 7) using instructions stored in memory hardware 720 (FIG. 7). The data processing hardware 710 and memory hardware 720 may be present in the user device 102 and / or the remote computing device 60 of FIG. 1 corresponding to the computing device 700 (FIG. 7).

[0048] At operation 602, the method 600 includes receiving a sequence of acoustic frames 110 corresponding to an utterance as an input to the personal VAD 300. At operation 604, the method 600 includes generating a reference speaker embedding 342 using a stack of multi-head self-attention blocks 340 of the personal VAD. At operation 606, the method 600 includes generating a speaker information embedding 312 extracted from the sequence of acoustic frames 110 using a speaker pre-net 310. At operation 608, the method 600 includes receiving a cosine similarity 322 between the target speaker embedding 412 and the speaker information embedding 312 as an input to the FiLM generator 330, and at operation 610, generating FiLM parameters 335 including a scaling vector 332 and a shift vector 334 based on the cosine similarity 322. That is, the FiLM generator 330 generates the FiLM parameters 335 based on the cosine similarity 322 instead of based on the target speaker embedding 412 (FIG. 5). At operation 612, the method 600 includes generating an affine transformed output 352 that scales (e.g., by the scaling vector 332) and shifts (e.g., by the shift vector 334) the reference speaker embedding based on the FiLM parameters 335. At operation 614, the method 600 includes generating a classification output 362 indicating whether the utterance was spoken by the target speaker based on the affine transformed output 352.

[0049] 7 is a schematic diagram of an exemplary computing device 700 that may be used to implement the systems and methods described herein. Computing device 700 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed herein.

[0050] Computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 that connects to memory 720 and high-speed expansion port 750, and a low-speed interface / controller 760 that connects to low-speed bus 770 and storage device 730. Each of the components 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be mounted on a common motherboard or otherwise as desired. Processor 710 can process instructions for execution within computing device 700, including instructions stored in memory 720 or storage device 730, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 780 coupled to high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and types of memory. Also, multiple computing devices 700 may be connected, with each device providing a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0051] The memory 720 stores information non-transiently within the computing device 700. The memory 720 may be a computer readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0052] The storage device 730 can provide mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various different implementations, the storage device 730 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as the methods described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 720, the storage device 730, or a memory on the processor 710.

[0053] The high-speed controller 740 manages bandwidth-intensive operations of the computing device 700, and the low-speed controller 760 manages lower bandwidth-intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 740 is coupled to the memory 720 (e.g., via a graphics processor or accelerator), the display 780, and is coupled to a high-speed expansion port 750 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to the storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and may be coupled, for example, via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0054] The computing device 700 may be implemented in a number of different forms, as shown: For example, the computing device 700 may be implemented as a standard server 700a, multiple times within a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0055] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0056] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0057] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose processors, as well as any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0058] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, speech input, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, such as, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0059] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A personal voice activity detector (VAD) (300), comprising: A stack of multi-headed self-attention blocks (340), receiving as input a sequence of acoustic frames (110) corresponding to an utterance; as an output, generating a reference speaker embedding (342) for the utterance; A stack of multi-headed self-attention blocks (340) configured to: A feature-wise linear modulation (FiLM) generator (330), comprising: receiving as input a target speaker embedding (412) for a target speaker; As an output, generating FiLM parameters (335) including a scaling vector (332) and a shift vector (334) based on the target speaker embedding (412). A FiLM generator (330) configured to: A FiLM layer (350), receiving as input the reference speaker embeddings (342) and the FiLM parameters (335); generating, as an output, an affine transformation output (352) that scales and shifts the reference speaker embedding (342) based on the FiLM parameters (335); A FiLM layer (350) configured to: a classifier (360) configured to generate a classification output (362) indicative of whether the utterance was spoken by the target speaker based on the affine transformation output (352); A personal voice activity detector (VAD) 300 comprising:

2. The classification output (362) target speaker token, Non-target speaker tokens, or Non-speech tokens The personal VAD (300) of claim 1, comprising at least one of:

3. A speaker pre-net (310), receiving as input said sequence of acoustic frames (110); generating, as an output, speaker information embeddings (312) extracted from the sequence of acoustic frames (110); 3. The personal VAD (300) of claim 1 or 2, further comprising a speaker pre-net (310) configured to:

4. The FiLM generator (330) receiving as input a cosine similarity (322) between the target speaker embedding (412) and the speaker information embedding (312); generating, as an output, the FiLM parameters (335) including the scaling vector (332) and the shift vector (334) based on the cosine similarity (322); The personal VAD (300) of claim 3, further configured to:

5. 5. The personal VAD (300) of claim 3 or 4, wherein the speaker pre-net (310) comprises a stack of multi-head self-attention layers including one or more conformer layers.

6. The personal VAD (300) of any of claims 1 to 5, wherein the stack of multi-headed self-attention blocks (340) comprises one or more conformer layers.

7. The personal VAD (300) of any of claims 1 to 6, wherein the classifier (360) comprises a fully connected layer.

8. A personal VAD (300) according to any of the preceding claims, wherein said personal VAD (300) operates in a streaming manner.

9. A pre-trained text-independent speaker recognition model (410), receiving as input an enrollment utterance (402) spoken by the target speaker; As an output, generating a target speaker embedding (412) for the target speaker based on the enrollment utterance (402). The personal VAD (300) of any of claims 1 to 8, further comprising a pre-trained text-independent speaker recognition model (410) configured to:

10. The personal VAD (300), enrollment training utterances paired with the target speaker embeddings (412); and unenrolled training utterances that are not paired with any corresponding target speaker embedding. The personal VAD (300) of any of claims 1 to 9, trained with training data comprising:

11. A computer-implemented method (500), when executed by data processing hardware (710), causing the data processing hardware (710) to: receiving a sequence of acoustic frames (110) corresponding to speech as input to a personal voice activity detector (VAD) (300); generating a reference speaker embedding (342) for the utterance using a stack of multi-head self-attention blocks (340) of the personal VAD (300); receiving a target speaker embedding (412) of a target speaker as an input to a feature-by-feature linear modulation (FiLM) generator (330) of the personal VAD (300); generating FiLM parameters (335) including a scaling vector (332) and a shift vector (334) based on the target speaker embedding (412) using the FiLM generator (330); using an FiLM layer (350) of the personal VAD (300) to generate an affine transformed output (352) that scales and shifts the reference speaker embeddings (342) based on the FiLM parameters (335); generating a classification output (362) indicating whether the utterance was spoken by the target speaker based on the affine transformation output (352) using a classifier (360) of the personal VAD (300); A computer-implemented method (500) for performing operations including:

12. The classification output (362) target speaker token, Non-target speaker tokens, or Non-speech tokens The computer-implemented method (500) of claim 11, comprising at least one of:

13. 13. The computer-implemented method (500) of claim 11 or 12, wherein the operations further comprise: generating a speaker information embedding (312) extracted from the sequence of acoustic frames (110) using a speaker pre-net (310) of the personal VAD (300).

14. 14. The computer-implemented method (500) of claim 13, wherein the operations further comprise: generating, using the FiLM generator (330), the FiLM parameters (335) based on a cosine similarity (322) between the target speaker embedding (412) and the speaker information embedding (312).

15. 15. The computer-implemented method (500) of claim 13 or 14, wherein the speaker pre-net (310) comprises a stack of multi-head self-attention layers including one or more conformer layers.

16. The computer-implemented method (500) of any of claims 11 to 15, wherein the stack of multi-headed self-attention blocks (340) comprises one or more conformer layers.

17. The computer-implemented method (500) of any of claims 11 to 16, wherein the classifier (360) comprises a fully connected layer.

18. The computer-implemented method (500) of any of claims 11 to 17, wherein the personal VAD (300) operates in a streaming manner.

19. The operation, receiving an enrollment utterance (402) spoken by the target speaker as an input to a pre-trained text-independent speaker recognition model (410); generating a target speaker embedding (412) for the target speaker based on the enrollment utterances (402) using the pre-trained text-independent speaker recognition model (410); The computer-implemented method (500) of any of claims 11 to 18, further comprising:

20. The personal VAD (300), enrollment training utterances paired with the target speaker embeddings (412); and unenrolled training utterances that are not paired with any corresponding target speaker embedding. The computer-implemented method (500) of any of claims 11 to 19, wherein the method is trained with training data comprising:

Images