Speaker verification using multitask speech models

Abstract

The method (400) includes obtaining a speaker identification (SID) model (151) trained to predict speaker embeddings (155) from utterances spoken by different speakers, the SID model including a trained audio encoder (150) and a trained SID head (154). The method also includes receiving a plurality of synthetic speech detection (SSD) training utterances (330) including a set of human spoken speech samples and a set of synthetic speech samples. The method also includes training an SSD head (160) with the SSD training utterances using the trained audio encoder to learn to detect the presence of synthetic speech in audio encodings encoded by the trained audio encoder. The operations also include providing a multitasking neural network model (130) for execution on the computing device (102) to perform both SID and SSD tasks in parallel on the input audio data.

Description

[Technical field]

[0001] This disclosure relates to automatic speaker verification using multi-task speech models. [Background technology]

[0002] In a voice-enabled environment, such as a home or automobile, a user can use voice input to access information and / or control various functions. The information and / or functions may be personalized for a given user. Therefore, it may be advantageous to identify a given speaker before allowing the user access to virtual assistants and telephone voice interfaces. Speaker identification (e.g., speaker verification and voice authentication) provides a simple way for registered users of a user device to access the user device. Speaker identification allows registered users to unlock and access the user's device by speaking, without having to manually enter a passcode (e.g., via typing or speaking) to access the user device. Speaker verification also allows digital assistants to identify registered users from spoken utterances without requiring the user to provide authentication credentials. In some cases, a would-be attacker may use a speech synthesis algorithm to create a synthesized voice in the voice of a registered user and trick the speaker identification system into authenticating the synthesized voice to access the user's device or invoke the digital assistant. In an additional example, a would-be attacker could record audio of a registered user speaking a verification phrase and then play the recorded audio to gain access to the user's device. As a result, systems that receive speech within an environment need some way to distinguish between human voice speech directed to the system and synthesized voice speech. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include obtaining a speaker identification (SID) model trained to predict speaker embeddings from utterances spoken by various speakers. The SID model includes a trained audio encoder and a trained SID head. The operations also include receiving a plurality of synthetic speech detection (SSD) training utterances including a set of human spoken speech samples and a set of synthetic speech samples. The operations also include training an SSD head with the SSD training utterances using the trained audio encoder to learn to detect the presence of synthetic speech in an audio encoding encoded by the trained audio encoder, the SSD head being overlaid on the trained audio encoder. The operations also include providing a multi-task neural network model for performing both SID and SSD tasks in parallel on input audio data for execution on a computing device. Here, the multi-task neural network model includes a trained audio encoder, a trained SID head overlaid on the trained audio encoder, and a trained SSD head overlaid on the trained audio encoder.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the SID model is trained with multiple SID training datasets, each including multiple SID training utterances spoken by M different speakers, where one or more SID training utterances spoken by a particular speaker may be paired with a corresponding target speaker embedding for training the audio encoder and SID head to learn how to predict a speaker embedding corresponding to the particular speaker. Further, in these implementations, the multiple SID training utterances of the one or more SID training sets may include N utterances per speaker among the M different speakers. Optionally, the multiple SID training utterances of each SID training dataset may include human-generated non-synthetic speech.

[0005] In some examples, the number of speech samples in the set of human-spoken speech samples is greater than the number of speech samples in the set of synthetic speech samples. In additional examples, the parameters / weights of the trained audio encoder are frozen while training the SSD head with the SSD training utterances. The audio encoder may include a multi-head attention model architecture. For example, the multi-head attention model architecture may incorporate a stack of conformer or transformer blocks. The SSD head may include one or more fully connected layers.

[0006] In some implementations, the multi-task neural network model is configured to perform SID and SSD tasks in parallel on the input audio data by receiving input audio data as input to a trained audio encoder, generating a shared audio encoding that encodes the input audio data as output from the trained audio encoder, providing the generated shared audio encoding as output from the trained audio encoder as input to a trained SID head and an SSD head, generating an evaluation speaker embedding that represents vocal characteristics of the speech characterized by the input audio data as output from the trained SID head, and generating an SSD classification score that indicates whether the speech characterized by the input audio data includes authentic human-generated speech or synthetic speech as output from the trained SSD head. The audio input data characterizes speech captured by an array of one or more microphones. In these implementations, the multi-task neural network model may be configured to grant access to one or more resources when the evaluation speaker embeddings match a reference speaker embedding associated with an enrolled user approved to access the resource and when the SSD classification scores generated as output from the trained SSD head indicate that the speech characterized by the input audio data includes authentic human-generated speech. Additionally or alternatively, in these implementations, the multi-task neural network model may be configured to restrict access to one or more resources when at least one of the evaluation speaker embeddings does not match any reference speaker embedding associated with an enrolled user approved to access the resource or when the SSD classification scores generated as output from the trained SSD head indicate that the speech characterized by the input audio data includes synthetic speech.

[0007] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware storing instructions executable on the data processing hardware to cause the data processing hardware to perform operations. The operations include obtaining a speaker identification (SID) model trained to predict speaker embeddings from utterances spoken by various speakers. The SID model includes a trained audio encoder and a trained SID head. The operations also include receiving a plurality of synthetic speech detection (SSD) training utterances including a set of human spoken speech samples and a set of synthetic speech samples. The operations also include training an SSD head with the SSD training utterances using the trained audio encoder to learn to detect the presence of synthetic speech in audio encodings encoded by the trained audio encoder, the SSD head being overlaid on the trained audio encoder. The operations also include providing a multitasking neural network model for performing both SID and SSD tasks in parallel on input audio data for execution on a computing device. Here, the multi-task neural network model includes a trained audio encoder, a trained SID head overlaid on the trained audio encoder, and a trained SSD head overlaid on the trained audio encoder.

[0008] This aspect of the disclosure may include one or more of the following optional features: In some implementations, the SID model is trained with multiple SID training datasets, each including multiple SID training utterances spoken by M different speakers, where one or more SID training utterances spoken by a particular speaker may be paired with a corresponding target speaker embedding for training the audio encoder and SID head to learn how to predict a speaker embedding corresponding to the particular speaker. Further, in these implementations, the multiple SID training utterances of the one or more SID training sets may include N utterances per speaker among the M different speakers. Optionally, the multiple SID training utterances of each SID training dataset may include human-generated non-synthetic speech.

[0009] In some examples, the number of speech samples in the set of human-spoken speech samples is greater than the number of speech samples in the set of synthetic speech samples. In additional examples, the parameters / weights of the trained audio encoder are frozen while training the SSD head with the SSD training utterances. The audio encoder may include a multi-head attention model architecture. For example, the multi-head attention model architecture may incorporate a stack of conformer or transformer blocks. The SSD head may include one or more fully connected layers.

[0010] In some implementations, the multi-task neural network model is configured to perform SID and SSD tasks in parallel on the input audio data by receiving input audio data as input to a trained audio encoder, generating a shared audio encoding that encodes the input audio data as output from the trained audio encoder, providing the generated shared audio encoding as output from the trained audio encoder as input to a trained SID head and an SSD head, generating an evaluation speaker embedding that represents vocal characteristics of the speech characterized by the input audio data as output from the trained SID head, and generating an SSD classification score that indicates whether the speech characterized by the input audio data includes authentic human-generated speech or synthetic speech as output from the trained SSD head. The audio input data characterizes speech captured by an array of one or more microphones. In these implementations, the multi-task neural network model may be configured to grant access to one or more resources when the evaluation speaker embeddings match a reference speaker embedding associated with an enrolled user approved to access the resource and when the SSD classification scores generated as output from the trained SSD head indicate that the speech characterized by the input audio data includes authentic human-generated speech. Additionally or alternatively, in these implementations, the multi-task neural network model may be configured to restrict access to one or more resources when at least one of the evaluation speaker embeddings does not match any reference speaker embedding associated with an enrolled user approved to access the resource or when the SSD classification scores generated as output from the trained SSD head indicate that the speech characterized by the input audio data includes synthetic speech.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram of an exemplary audio environment having speaker identification and a synthetic voice detection system for identifying a speaker; [Diagram 2] 1 is a schematic diagram of an example of conformer blocks that may form a stack of conformer layers implemented by an encoder of a speaker identification and synthetic voice detection system. [Figure 3A] FIG. 1 is a schematic diagram of a first training stage, in which an untrained audio encoder and speaker identification (SID) head are trained on a SID task to learn how to predict speaker embeddings / representations from input audio. [Figure 3B] FIG. 13 is a schematic diagram of a second training stage for training a synthetic speech detection (SSD) head to learn how to predict whether input audio corresponds to authentic human-produced speech or synthetic / synthesized speech. [Figure 4] 1 is a flowchart of an exemplary operational arrangement of a method for training a multi-tasking neural network model to perform both SID and SSD tasks in parallel on input audio data. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference symbols in the various drawings indicate like elements.

[0014] In a voice-enabled environment, such as a home, a car, a workplace, or a school, a user may speak a query or command, causing a digital assistant to answer the query and / or perform the command. Such a voice-enabled environment may be implemented with a network of connected microphone devices distributed in various rooms or areas of the environment. Through the network of microphones, a user may query a digital assistant through spoken utterances without the need to have a computer or other interface in front of them. In some cases, the voice-enabled environment is associated with multiple registered users, e.g., people living in a household. These cases may be the case when a single device is shared by multiple users, such as a smartphone, a smart speaker, a smart display, a tablet device, a smart TV, a smart appliance, a vehicle infotainment system, etc. It is therefore desirable to determine the identity of a particular user speaking a query. The process of determining the identity of a particular speaker / user may be referred to as speaker verification, speaker recognition, speaker identification, or voice recognition.

[0015] Speaker verification / identification may enable a user to issue queries that act on behalf of a particular user and / or trigger personalized responses in a multi-user environment. Speaker verification / identification (e.g., voice authentication) provides a simple way for a user of a user device to gain access to the user device. For example, a user may unlock and access the user device by speaking an utterance, without the user having to manually enter (e.g., via typing or speaking) a passcode to gain access to the user device.

[0016] In some scenarios, a user queries a digital assistant that requires access to resources from a set of personal resources related to the user's personal information and / or associated with the user. For example, a particular user (e.g., a user who is registered with the digital assistant) may ask the digital assistant "When is my meeting with Matt?" or query the digital assistant "Play my music playlist." Here, the user may be one of one or more registered users, each with permission to access their own respective set of personal resources (e.g., calendar, music player, email, messaging, contact list, etc.) and restricted from accessing the personal resources of other registered users. For example, if John is a registered user of the digital assistant, the digital assistant needs to distinguish whether John spoke the utterance "When is my meeting with Matt" in order to access the calendar of the appropriate registered user of John to determine when a meeting with Matt is scheduled and respond with the meeting details of the scheduled meeting with Matt. Similarly, the digital assistant needs to distinguish if John said "Play my music playlist" in order to access the music player and ultimately audibly output a track from John's music playlist.

[0017] To determine which user is speaking in a multi-user voice-enabled environment, the voice-enabled system may include a speaker identification system (e.g., a speaker verification system or a voice authentication system). An automatic speaker verification (ASV) system may employ a speaker identification (SID) model trained to extract rating vectors (e.g., speaker embeddings) from audio data corresponding to utterances spoken by a particular user, such that the extracted rating vector represents the voice characteristics of the particular user. The rating vector may include a d-vector. The SID model may be a neural network model trained under machine or human supervision to output the d-vector. To resolve the identity of a particular user, a comparator determines whether the rating vector matches a reference vector of any of the enrolled and / or authorized users of the user device, where each reference vector may correspond to a voiceprint or unique identifier representing the voice characteristics of the respective enrolled / authorized user. Each enrolled and / or approved user can speak multiple enrollment utterances, and for each enrollment phrase, the SID model may generate a corresponding reference d-vector, which may be combined, e.g., averaged, or otherwise accumulated to form a respective reference d-vector for the enrolled / approved user.

[0018] With the advent of synthetic media (e.g., "deep fakes"), it is crucial for these systems to accurately determine when a voice utterance contains synthetic speech (i.e., computer-generated audio output that resembles a human voice). For example, state-of-the-art text-to-speech (TTS) and voice conversion (VC) systems can now closely mimic human speakers, providing a means to attack and impersonate ASV systems. Thus, state-of-the-art TTS and VC systems can be used to generate synthesized / synthetic speech representations in the voice of an authorized / registered user and spoof the SID model to verify that the synthesized speech representation was spoken by the authorized / registered user. As used herein, the terms "synthesized voice," "synthetic voice," "synthetic speech," and "synthesized speech" are used interchangeably. Impostors may also play audio of a pre-recorded voice of a registered user to impersonate an ASV system to gain unauthorized access to the registered user's locked user device and / or resources associated with the registered user. As a result, the microphone on the user device may capture synthesized speech or played audio of the registered user's pre-recorded voice, and the ASV system may extract an evaluation vector that matches the registered user's reference vector, thus impersonating the ASV system and allowing the impostor to gain unauthorized access to the user device.

[0019] To detect these impersonation attempts, the ASV system may include one or more countermeasure systems for synthetic voice detection (SSD) and / or replay attack detection (RAD). Because SID models are typically neural network based, conventional countermeasure systems for SSD and RAD are also typically based on neural network models trained via machine learning techniques. Thus, employing additional countermeasure systems in conjunction with the SID model may significantly increase computation / storage requirements due to the need to run multiple separate neural network models in parallel, i.e., separate neural network models for each of SID, SSD, and RAD on the input audio signal. Since the primary use case for ASV systems is to authenticate users "in real time" to launch virtual assistants and / or gain access to the phone's voice interface, minimizing latency is crucial to ensure a positive user experience. Furthermore, because deployment of ASV systems typically occurs on resource-constrained systems (e.g., smartphones), it is important to minimize computation and storage resource usage.

[0020] Machine learning (ML) algorithms, such as neural networks, are primarily driving the proliferation of ASV systems and other voice-enabled technologies. However, these algorithms traditionally require large amounts of training samples, and as a result, the main bottleneck in training accurate models is often the lack of sufficiently large, high-quality datasets. For example, while large datasets containing human-uttered speech are readily available, similar datasets containing synthetic speech instead are not. Thus, training a model that can accurately determine synthetic speech without a traditional training set poses a significant challenge to the development of synthetic speech detection systems.

[0021] Embodiments herein are directed to a multi-task neural network model that can simultaneously perform both speaker identification and synthetic speech detection tasks without the need to employ separate neural network models to perform the SID and SSD tasks. In some embodiments, the multi-task neural network model performs replay attack detection (RAD) in addition to or instead of the SSD task. Among other things, the multi-task neural network model includes an audio encoder that receives as an input an input audio signal that characterizes an utterance and generates as an output a shared output embedding for use in at least one of the SID head and the SSD head or the RAD head. Here, each of the SID head, SSD head, and RAD head may be layered on top of the audio encoder and configured to compute outputs associated with each of the SID task, SSD task, and RAD task. For example, the SID head may include a fully connected network that receives the shared output embedding generated by the audio encoder and computes a speaker embedding that characterizes the voice characteristics of the speaker of the utterance. In this manner, the audio encoder and the SID head may collectively provide a SID model. The SSD head may also receive the same shared output embeddings generated by the audio encoder, but instead may include a fully connected network that calculates a synthetic speech score indicating whether the input audio signal corresponds to authentic speech (e.g., non-synthetic / real speech / human speech) or synthetic speech. In this way, the audio encoder and the SSD head may collectively provide the SSD model. If applicable, the RAD head may include a fully connected network that also receives the same shared output embeddings generated by the audio encoder, and calculates a RAD score indicating whether the input audio signal corresponds to pre-recorded speech played from an audio output device or unrecorded speech audibly spoken by a human. In this way, the audio encoder and the SSD head may collectively provide the RAD model.

[0022] As described in more detail below, the audio encoder and SID head may be initially trained on SID training utterances spoken by different speakers to learn how to predict speaker embeddings for input audio data. The trained audio encoder may then be used to train the SSD head and / or RAD head to learn how to predict the aforementioned synthetic speech and / or RAD scores. The initial training process may first train a multi-task neural network model on a SID task to teach the multi-task neural network model to learn to generate speaker embeddings from SID training utterances spoken by different speakers.

[0023] 1, in some implementations, an exemplary system 100 includes user devices 102 associated with one or more users 10 and communicating with a remote system 111 via a network 104. The user devices 102 may correspond to computing devices such as mobile phones, computers (laptop or desktop), tablets, smart speakers / displays, smart appliances, smart headphones, wearables, vehicle infotainment systems, etc., and include data processing hardware 103 and memory hardware 107. The user devices 102 may include or communicate with one or more microphones 106 for capturing speech from the respective users 10. The remote system 111 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources 113 (e.g., data processing hardware) and / or storage resources 115 (e.g., memory hardware).

[0024] The user device 102 may include a hot word detector (not shown) configured to detect the presence of hot words in the streaming audio 118 without performing semantic analysis or speech recognition processing on the streaming audio 118. The user device 102 may include an acoustic feature extractor (not shown) that may be implemented as part of the hot word detector or as a separate component for extracting audio data 120 from the utterance 119. For example, with reference to FIG. 1, the acoustic feature extractor may receive streaming audio 118 captured by one or more microphones 106 of the user device 102 corresponding to the utterance 119 spoken by the user 10 and extract the audio data 120. The audio data 120 may include acoustic features such as Mel-frequency cepstral coefficients (MFCCs) or filter bank energies calculated over a window of the audio signal. In the illustrated example, the utterance 119 spoken by the user 10 includes "Ok Google, play my music playlist."

[0025] The hot word detector may receive the audio data 120 and determine whether the utterance 119 includes a particular hot word (e.g., Ok Google) spoken by the user 10. That is, the hot word detector 110 may be trained to detect the presence of a hot word (e.g., Ok Google) or one or more variants of a hot word (e.g., Hey Google) in the audio data 120 and cause the user device 102 to wake up from a sleep or hibernation state and trigger an automatic speech recognition (ASR) system 180 to perform voice recognition on the hot word, e.g., the hot word followed by a voice query specifying an action to be taken, and / or one or more other terms. In the illustrated example, the query following the hotword of the captured speech 119 in the streaming audio includes "Play my music playlist," which specifies an action for the digital assistant to access a music playlist associated with a particular user (e.g., John) 10, and provides a response 190 including an audio track from John's music playlist to the user device 102 (and / or one or more selected audio output devices) to play an audible output from a speaker.

[0026] The system 100 includes a multi-task neural network model 130 configured to process the audio data 120 to determine the identity of the user 10 speaking the utterance 119 and to determine whether the audio data 120 includes disguised speech. As used herein, the multi-task neural network model 130 may include determining that the audio data 120 includes disguised speech by detecting that the audio data 120 includes synthetic speech and / or, optionally, detecting that the audio data 120 includes pre-recorded speech played from an audio output device. Advantageously, the multi-task neural network model 130 includes a shared audio encoder 150 that receives the audio data 120 as an input and generates, as an output, an audio encoding 152 that corresponds to a hidden feature representation of the audio data 120 encoded by the audio encoder 150. Each audio encoding 152 output from audio encoder 150 may be input to each of SID head 154, SSD head 160, and RAD head 161 that branch off from audio encoder 150. Thus, audio encoder 150 may function as the "body" of multi-task neural network model 130, with audio encoder 150 and SID head 154 forming the SID branch corresponding to SID model 151 (FIG. 3A), audio encoder 150 and SSD head 160 forming the SSD branch corresponding to SSD model 153 (FIG. 3B), and audio encoder 150 and RAD head 161 forming the RAD branch corresponding to the RAD model.

[0027] The multi-tasking neural network model 130 may execute on a computing device, including the data processing hardware 103, of the user device 102 associated with the user 10, or the multi-tasking neural network model 130 may execute on the data processing hardware 113 of the remote system 111. When executing on the data processing hardware 113 of the remote system 111, the user device 102 may transmit audio data 120 corresponding to speech 119 captured by the user device 102 over the network 104 to the multi-tasking neural network model 130 executing on the remote system 111. In some implementations, components of the multi-tasking neural network model 130 are distributed for execution across the data processing hardware 103 of the user device 102 and the data processing hardware 113 of the remote system 111. For example, the audio encoder 150 may execute on one of the user device 102 or the remote system 111, while the SID, SSD, and RAD heads 154, 160, 161 execute on the other of the user device 102 or the remote system 111. Such a configuration may be beneficial when processing / storage / memory resources are limited on the user device 102. The multitasking neural network model 130 may employ all three of the SID, SSD, and RAD heads 154, 160, 161, or any combination of two of the heads. For example, the multitasking neural network model 130 may employ the SID head 154 and the SSD head 160, but not the RAD head 161.

[0028] In some implementations, audio encoder 150 includes a multi-head attention model architecture. For example, audio encoder 150 may include a stack of conformer blocks. The multi-head attention model architecture of audio encoder 150 need not be limited to conformer blocks, but may include any type of multi-head attention model architecture, including, but not limited to, a stack of transformer blocks or lightweight convolutional blocks.

[0029] FIG. 2 provides an example of a conformer block 200 from a stack of conformer layers of the encoder 150. The conformer block 200 includes a first half-feedforward layer 210, a second half-feedforward layer 240, and a concatenation operator 205, with a multi-head self-attention block 220 and a convolutional layer 230 disposed between the first half-feedforward layer 210 and the second half-feedforward layer 240. The first half-feedforward layer 210 processes audio features 121 extracted from the input audio data 120. For example, an acoustic front-end (not shown) may extract the audio features 121 from the input audio data 120 for input to the audio encoder. The audio features 121 may include a log-mel filter bank computed and stacked every four frames with a stride of three. The multi-head self-attention block 220 then receives the input audio data 120 concatenated with the output of the first half-feedforward layer 210. In particular, the multi-head self-attention block 220 integrates a relative sinusoidal position encoding scheme that allows the self-attention block 220 to generalize better to different input lengths, making the resulting encoder 150 more robust to variations in the length of the input audio data 120 that characterize the speech 119. Intuitively, the role of the multi-head self-attention block 220 is to separately summarize the acoustic context for each input frame to be encoded. The convolutional layer 230 subsamples the output of the multi-head self-attention block 220 concatenated with the output of the first half-feedforward layer 210. The second half-feedforward layer 240 then receives the concatenation of the output of the convolutional layer 230 and the multi-head self-attention block 220. The layernorm module 250 processes the output from the second half-feedforward layer 240. Mathematically, the conformer block 200 transforms the input feature x using the modulation feature m to generate the output feature y as follows:

number

[0030] Referring back to FIG. 1 , in some examples, the multi-task neural network model 130 is deployed in a multi-user voice-enabled environment in which multiple different users 10 are each registered with a user device 102 and have permission to access a respective set of personal resources associated with that user (e.g., calendar, music player, email, messaging, contact list, etc.). A registered user 10 is restricted from accessing personal resources from the respective sets of personal resources associated with other registered users. Each registered user 10 may have a respective user profile that links to the respective sets of personal resources associated with that user, as well as other relevant information associated with that user 10 (e.g., user-specified preference settings).

[0031] The audio encoder 150 and the SID head 154 may be initially trained on the SID task to determine an evaluation speaker embedding / vector 155 for each of the utterances 119 that conveys the voice characteristics of the user 10. The reference speaker embedding / vector 155 represents the voice characteristics of the utterances 119 captured by the user device 102. The verifier 170 may then compare the speaker embedding 155 to one or more reference speaker embeddings 171 associated with registered / approved users who have permission to access the respective set of personal resources (e.g., calendar, music player, email, messaging, contact list) associated with that user. The evaluation and reference speaker embedding / vector 155 may include a d-vector or an i-vector. When the speaker embedding / vector 155 matches one of the reference speaker embeddings 171, the verifier 170 may verify the identity of the user 10 who spoke the utterance 119 as one of the registered / approved users. Advantageously, the SID / ASV allows the user 10 to, among other things, unlock and access the user device 102 by speaking an utterance without the user having to manually enter (e.g., via typing) or speak a passcode or provide some other means of verification (e.g., answering a challenge question, providing biometric verification data, etc.) to gain access to the user device 102. Similarly, in a multi-user voice-enabled environment, the SID / ASV made by the audio encoder 150, SID head 154, and verifier 170 may be used to determine which registered user 10 is speaking the utterance 119. For example, in FIG. 1, John may be a registered user 10 of the user device 102 (or an interface for the digital assistant 101 running on the user device), and the digital assistant 101 needs to distinguish whether John spoke the utterance 119 "Ok Google, play my music list" in order to access the music player and ultimately audibly output a track from the appropriate music playlist associated with John.

[0032] The audio encoder 150 initially trained on the SID task may then be used to train the SSD head 160 on the SSD task to determine whether the input audio data 120 characterizing the utterance 119 includes synthetic speech. For example, the utterance 119 captured by the user device 102 in the streaming audio 118 may correspond to synthetic speech generated by a text-to-speech (TTS) system 11 or a voice conversion (VC) model 13. Here, the TTS system 11 may convert the input text corresponding to the utterance 119 into synthetic speech having voice characteristics of an authorized / registered user 10 having permission to access the user device 102. Similarly, the VC model 13 may receive audio of the utterance 119 spoken by a fake user not authorized to access the user device 102 and convert the audio into converted audio having voice characteristics of the authorized / registered user 10. Thus, a fake user / entity may employ the TTS system 11 and / or VC model 13 to generate a speaker embedding of the approved / registered user 10 by impersonating the SID head 154 in order to generate a synthetic voice having the voice characteristics of the approved / registered user to gain unauthorized access to a set of personal resources associated with the user device 102 and / or the approved / registered user 10. As a countermeasure to prevent impersonation of the verifier 170, the SSD head 160 of the multi-task neural network model 130 is configured to receive as an input the audio encoding 152 generated as an output by the audio encoder 150, and to generate as an output an SSD classification score 162 indicative of whether the audio data 120 characterizing the utterance 119 includes authentic human-generated speech or a synthetic / impersonated speech. In some examples, the SSD classification score 162 includes a probability score that indicates the presence of a synthesized voice when the probability score 162 meets an SSD threshold score. In these examples, the disguised voice detector 172 may compare the probability score conveyed by the SSD classification score 162 to an SSD threshold score to determine an output indicating whether the utterance 119 is authentic voice or synthetic / disguised voice.In other examples, the SSD classification score 162 comprises a binary / Boolean value indicating the presence of authentic human-generated speech when the value is equal to "0" or synthesized speech when the value is equal to "1", or vice versa.

[0033] Similar to SSD head 160, audio encoder 150 first trained on the SID task may then be used to train RAD head 161 on the RAD task to determine whether input audio data 120 characterizing speech 119 includes pre-recorded voice played from an audio output device. As previously described, the pre-recorded voice of the enrolled user may be played from the audio output device, and a microphone on user device 102 may capture the played audio of the pre-recorded voice of the enrolled user. SID head 154 may then extract evaluation vector 155 that verifier 170 verifies as a match to reference vector 171 for the enrolled user, thereby allowing the impostor to gain unauthorized access to the user device by impersonating verifier 170. As a countermeasure to prevent impersonation of the verifier 170, the RAD head 161 of the multi-task neural network model 130 is configured to receive as an input the audio encoding 152 generated as an output by the audio encoder 150 and generate as an output a RAD classification score 163 indicative of whether the audio data 120 characterizing the utterance 119 includes authentic speech associated with unrecorded speech audibly spoken by a human or disguised speech associated with pre-recorded speech played from an audio output device. In some examples, the RAD classification score 163 includes a probability score that indicates the presence of pre-recorded speech when the probability score 163 meets a RAD threshold score. In these examples, the disguised speech detector 172 may compare the probability score conveyed by the RAD classification score 163 to the RAD threshold score to determine an output indicative of whether the utterance 119 is authentic speech or pre-recorded / disguised speech. In other examples, the RAD classification score 163 includes a binary / Boolean value indicating the presence of authentic unrecorded human-generated speech when the value is equal to "0" or pre-recorded speech when the value is equal to "1", or vice versa.

[0034] In some implementations, the multi-task neural network model 130 is configured to restrict access to one or more resources when at least one of the verifier 170 determines that the evaluation speaker embedding 155 does not match any reference speaker embedding 171 associated with a registered user authorized to access the resource or when the impersonated voice detector 172 determines that the SSD classification score 162 and / or the RAD classification score 163 include synthetic and / or pre-recorded voice. In these implementations, the multi-task neural network model 130 prevents the ASR system 180 from performing speech recognition on the audio data 120 or otherwise inhibits speech recognition from being performed on the audio data 120 if the ASR system 180 has already started performing speech recognition.

[0035] In some additional implementations, the multi-task neural network model 130 is configured to approve access to one or more resources when the verifier 170 determines that the evaluation speaker embedding 155 matches a reference speaker embedding 171 associated with an enrolled user approved to access the resource, and when the impersonated voice detector 172 determines that the SSD classification score 162 and / or the RAD classification score 163 include authentic human-generated speech. In these implementations, the verifier 170 and the impersonated voice detector 172 may collectively output a confirmation 175 that, when received by the ASR system 180, causes the ASR system 180 to perform speech recognition on the audio data 120 characterizing the utterance 119. If the ASR system 180 has already begun performing speech recognition, the confirmation 175 causes the ASR system 180 to continue processing the audio data 120 to recognize terms spoken by the user 10 in the utterance 119.

[0036] The ASR system 180 may include an ASR model configured to perform speech recognition on the audio data 120 characterizing a query, e.g., “play my music playlist.” The ASR system 180 may also include a natural language understanding (NLU) module configured to perform query interpretation on the speech recognition results output by the ASR model. In general, the NLU module may perform semantic analysis on the speech recognition results to identify an action to perform specified by the query. In the illustrated example, the NLU module may determine that performing the action specified by the query “play my music playlist” requires access to a respective set of personal resources associated with each registered user 10 of the user device 102. Thus, the NLU module determines that the action specified by the query lacks a required parameter, i.e., the user's identity, required to perform the action. Thus, the NLU module may use the identity of a particular registered user (e.g., John) 10 verified by the verifier 170 to ascertain which resources to access to perform an action, and thus begin fulfilling the query by providing output instructions 185 to perform the action specified by the query. In the illustrated example, the output instructions 185 may instruct a music streaming service to stream music tracks from the registered user John's music playlist. The digital assistant 101 may provide a response to the query that includes music tracks for audible output from the user device 102 and / or one or more other devices in communication with the user device 102.

[0037] 3A and 3B show an example training process 300 for training a multi-task neural network model 130 to perform both speaker identification (SID) and synthetic speech detection (SSD) tasks simultaneously without having to employ separate neural network models for performing the SID and SSD tasks. The training process 300 may run on the remote system 111 of FIG. 1. The training process 300 includes a first training stage 300a (FIG. 3A) in which an untrained audio encoder 150 and a SID head 154 superimposed on the untrained audio encoder 150 are trained on the SID task to learn how to predict speaker embeddings / representations 155 from input audio. After the first training stage 300a is completed and the audio encoder 150 and SID head 154 are fully trained to perform an SID task (e.g., automatic speaker verification), the training process 300 includes a second training stage 300b (FIG. 3B) that uses the audio encoder 150 trained on the SID task by the first training stage 300a to train the SSD head 160 to learn how to predict whether input audio corresponds to authentic human-generated speech or synthetic / synthesized speech.

[0038] Referring to FIG. 3A, the first training stage 300a (i.e., the SID training stage) obtains one or more SID training data sets 310 stored in the data storage 301 and trains the untrained audio encoder 150 and the SID head 154 with the training data sets 310. Once trained by the first training stage 300a, the audio encoder 150 and the SID head 154 may collectively form the SID model 151. The data storage 301 may reside on the memory hardware 115 of the remote system 111. Each training data set 310 includes multiple training utterances 320, 320a-n spoken by M different speakers, allowing the training data set 310 to include negative training utterances 320 of multiple different speakers in a given training batch. Furthermore, each training data set 310 may include N utterances 320 per speaker among the M different speakers, allowing the training data set 310 to also include positive training utterances 320 of the same speaker in a given training batch. Each corresponding training utterance 320 may include a text-dependent portion 321 and / or a text-independent portion 322. The text-dependent portion 321 includes an audio segment that features the spoken predetermined word (e.g., "Hey Google") or a variant of a predetermined hot word (e.g., "Ok Google") of the training utterance 320. In additional implementations, the text-dependent portion 321 of some training utterances 320 includes an audio segment that features other terms / phrases instead of the predetermined word or variants thereof, such as a custom hot word, a verification phrase (e.g., "This is Meg"), or a commonly used voice command (e.g., play, pause, turn volume up / down, call, message, navigate / directions, etc.). The text-dependent portion 321 is optional such that only a portion of the training utterances 320 or none of the training utterances 320 may include a text-dependent portion 321.

[0039] In some examples, the text-independent portion 322 of each training utterance 320 includes an audio segment that characterizes a predetermined word characterized by the text-dependent portion 321 followed by a query statement spoken in the training utterance 320. For example, a corresponding training utterance 320 may include "Ok Google, what's the weather outside?", thereby including a text-dependent portion 321 that characterizes the predetermined "Ok Google" and a text-independent portion 322 that characterizes the query statement "What's the weather outside?". While the text-dependent portion 321 in each training utterance 320 is phonetically constrained by the same predetermined word or variations thereof, the lexicon of the query statements characterized by each text-independent portion 322 is not constrained, such that the durations and phonemes associated with each query statement are variable.

[0040] With continued reference to FIG. 3A , the first stage 300a of the training process 300 trains an untrained audio encoder 150 on training utterances 320, 320a-n to generate, as output from the audio encoder 150, audio encodings 152 for the training utterances 320. That is, the audio encoder 150 is configured to encode audio data characterizing the training utterances into a hidden feature representation (e.g., a series of encodings / vectors 152). The SID head 154 receives as input the audio encodings 152 output from the audio encoder 150 and generates as output a respective speaker embedding 155 for each utterance 320. In some examples, the speaker embeddings 155 output by the SID head 154 for each utterance include d-vectors. However, the first training stage 300a may also train the audio encoder 150 and the SID head 154 on the SID task to generate other types of speaker embeddings 155, such as i-vectors. During training, additional information about each utterance 320 may be provided as input to an SID model 151, which consists of an audio encoder 150 and an SID head 154 overlaid on the audio encoder 150. For example, SID targets 324, such as SID target vectors / speaker embeddings, corresponding to ground truth output labels for training the SID model 151 to learn how to predict speaker embeddings 155 for a particular speaker, may be provided as input to the neural network during training with the utterances 320. Thus, one or more utterances 320 from each particular speaker may be paired with a particular SID target vector 324.

[0041] As mentioned above, the audio encoder 150 includes a multi-head attention model architecture that may incorporate a stack of conformer blocks. Optionally, other types of multi-head attention model architectures, such as, but not limited to, transformer blocks or lightweight convolutional blocks, instead of conformer blocks, may be employed in the audio encoder 150. The SID head 154 stacked on the audio encoder 150 may include one or more fully connected network layers selected to train to generate speaker-related information specific to the speaker embeddings 155.

[0042] Training the untrained audio encoder 150 and the SID head 154 may begin by using a pair-wise training technique, where a sequence of training utterances 320 is provided, where first and second training utterances 320, each paired with a particular SID target vector 324, are input to the audio encoder 150, and the corresponding audio encodings 152 generated by the audio encoder 150 are processed by the SID head 154 to generate respective speaker embeddings 155 for each of the first and second training utterances 320. The SID loss module 156 may then compare the respective speaker embeddings 155 to determine whether the two speaker embeddings 155 are derived from training utterances 320 uttered by the same speaker. That is, the SID loss module 156 may calculate a SID loss 158 indicating the likelihood that the two speaker embeddings 155 match each other. The SID loss 158 output from the SID loss module 156 provides an indication of whether the training utterances of the first and second training utterances 320 were spoken by the same speaker. In one example, the SID loss 158 may simply include a binary value of "0" or "1," with "0" indicating that the utterances were spoken by different speakers and "1" indicating that the utterances were spoken by the same speaker. Parameters of the audio encoder 150 and the SID head 154 may then be adjusted based on the SID loss 158. Multiple sets of pair-wise training utterances 320 may be processed in this manner.

[0043] 3B, the second training stage 300b (i.e., SSD training stage) receives / obtains the SID model 151 trained by the first training stage 300a and uses the trained audio encoder 150 to train the SSD head 160 to learn how to predict whether an input audio corresponds to authentic human-generated speech or synthetic / synthetic speech. During the second training stage 300b, the parameters / weights of the trained audio encoder 150 are fixed and the SSD head 160 is trained (or fine-tuned) to learn how to project an SSD classification score 162 indicating whether the input audio contains authentic human-generated speech or synthetic speech. As will become clear, the SSD head 160 receives as input the audio encodings 152 generated by the trained audio encoder 150 for the SSD training utterances 330 and generates as output a respective SSD classification score 162 for each SSD training utterance 330. In some examples, the SSD classification score includes a probability score 162 indicating the presence of synthesized speech when the probability score 162 meets an SSD threshold. In other examples, the SSD classification score includes a binary / Boolean value indicating authentic human generated speech when the value is equal to "0" or the presence of synthesized speech when the value is equal to "1", or vice versa.

[0044] Similar to the SID head 154 employed in the SID model 151, the SSD head 160 layered / superimposed on the trained audio encoder 150 may include one or more fully connected network layers. However, in contrast to the fully connected network layers employed in the SID head 154, the fully connected network layers of the SSD head 160 are trained / selected to learn to project the SSD classification scores 162. In some examples, the second training stage 300b trains one or more fully connected network layers of the SSD head 160 from scratch using the SSD training utterances 330. In other examples, the one or more fully connected network layers of the SSD head 160 include the same fully connected network layers of the SID head 154 trained by the first training stage 300a, and the second training stage 300b fine-tunes the one or more fully connected network layers to learn how to project the speaker embeddings 155 into the SSD classification scores 162 with the SSD training utterances 330 for use by the SSD head 160. Notably, regardless of whether the SSD head 160 is fine-tuned or trained from scratch during the second training stage 300b, the parameters / weights of the trained audio encoder 150 remain fixed while the SSD training utterances 330 are used to train the SSD head 160 on the SSD task.

[0045] The second training stage 300b receives / obtains a plurality of SSD training utterances 330 stored in the data storage 301 and uses the trained audio encoder 150 to train the SSD head 160 with the SSD training utterances 330. When trained by the second training stage 300b, the audio encoder 150 and the SSD head 160 may collectively form the SSD model 153. The data storage 301 may reside on the memory hardware 115 of the remote system 111. The SSD training utterances 330 may include a set of human-spoken speech samples 330a and a set of synthetic speech samples 330b. In some examples, the number of speech samples in the set of human-spoken speech samples 330a is greater than the number of speech samples in the set of synthetic speech samples 330b. In other examples, the number of speech samples in the set of human-spoken speech samples 330a is less than the number of speech samples in the set of synthetic speech samples 330b. Each human-spoken voice sample 330a corresponds to a real / human spoken non-synthetic voice utterance, and each synthetic voice sample 330b corresponds to a machine-generated synthetic / synthetic voice. That is, one or more of the synthetic voice samples 330b may include a synthesized voice generated by a text-to-speech (TTS) system from a corresponding input text. Additionally or alternatively, one or more of the synthetic voice samples 330b may include a synthesized voice generated by a voice conversion model from a corresponding input utterance having different voice characteristics than the generated synthesized voice.

[0046] In some implementations, the second training stage 300b trains the SSD head 160 using supervised learning, where each SSD training utterance 330 is paired with a corresponding ground truth target 165 that indicates whether the SSD training utterance 330 includes a human-generated speech sample 330a or a synthetic speech sample 330b. Here, the SSD loss module 168 may apply a loss function to the classification scores 162 predicted by the SSD head 160 to generate a corresponding SSD loss 169 based on the ground truth target 165. In some examples, the loss function includes a softmax cross-entropy loss, where a softmax layer applies a softmax to the classification scores 162 (e.g., Boolean values) output from the SSD head 160 that are fed into the cross-entropy loss function to generate the SSD loss 169.

[0047] In other embodiments, the second training stage 300b trains the SSD head 160 using self-supervised learning (also called "unsupervised learning"), where the labels are a natural part of the training utterances 330 and do not include separate external ground truth labels, i.e., each human-generated speech sample 330a does not include a separate external ground truth label indicating that the sample contains authentic speech, and each synthetic speech sample 330b does not include a separate external ground truth label indicating that the sample contains synthesized speech. Here, the human-generated speech samples 330a may correspond to positive training samples, and the synthetic speech samples 330b may correspond to negative training samples where the SSD model 153 learns to look for patterns without pre-existing annotated labels and with minimal human supervision.

[0048] The third training stage may similarly train the RAD head 161 using the trained audio encoder 150 trained during the first training stage 300a of FIG. 3A. The third training stage trains the RAD head 161 to learn how to predict whether an input audio corresponds to authentic non-prerecorded, human-generated speech or pre-recorded speech played from an audio output device. Similar to the second training stage 300b, the parameters / weights of the trained audio encoder 150 are fixed and the RAD head 161 is trained (or fine-tuned) to learn how to project a RAD classification score 163 indicative of whether the input audio contains authentic speech or disguised / pre-recorded speech. The RAD classification score 163 may include a probability distribution indicating the likelihood that the audio data contains pre-recorded speech, or may include a Boolean / binary value indicating the presence of authentic non-prerecorded, human-generated speech when the value is equal to "0" or pre-recorded speech when the value is equal to "1", or vice versa.

[0049] In some examples, SSD head 160 is also trained to detect the presence of pre-recorded speech in the input audio, thereby mitigating the need for a separate RAD head 161. In this manner, SSD head 160 may be trained to output a classification score indicative of input audio containing authentic speech or disguised speech when either one of synthetic speech or pre-recorded speech is detected.

[0050] 4 provides an exemplary flowchart of an exemplary arrangement of operations of a method 400 for training a multi-task neural network model 130 to perform both SID and SSD tasks in parallel on input audio data. The method 400 may execute on the data processing hardware 113 of the remote system 111 and / or on the data processing hardware 103 of the user device 102. In operation 402, the method includes obtaining a speaker identification (SID) model 151 trained to predict speaker embeddings 155 from utterances spoken by different speakers. The SID model 151 includes a trained audio encoder 150 and a trained SID head 154. The trained audio encoder 150 may include a multi-head attention model architecture. For example, the multi-head attention model architecture may incorporate a stack of conformer blocks 200. In other configurations, the multi-head attention model architecture may incorporate a stack of transformer blocks or a stack of lightweight convolution blocks.

[0051] The obtained SID model 151 may be trained with multiple SID training data sets 320, each including multiple SID training utterances spoken by M different speakers, where one or more of the SID training utterances spoken by a particular speaker may be paired with a corresponding target speaker embedding 324 for training the audio encoder 150 and the SID head 154 to learn how to predict the corresponding speaker embedding 155 of the particular speaker. Additionally or alternatively, the multiple SID training utterances in one or more of the SID training sets 320 may include N utterances spoken per speaker among the M different speakers. Furthermore, the multiple SID training utterances in each SID training set 320 may include non-synthetic human-spoken speech.

[0052] At operation 404, the method 400 includes receiving a plurality of SSD training utterances 330, where the SSD training utterances 330 include a set of human-spoken speech samples 330a and a set of synthetic speech samples 330b, where in some examples, the number of speech samples in the human-spoken speech samples 330a is greater than the number of speech samples in the set of synthetic speech samples 330b.

[0053] At operation 406, the method 400 includes training an SSD head 160 with the SSD training utterances 330 to learn to detect the presence of synthetic speech in the audio encodings 152 encoded by the trained audio encoder 150 using the trained audio encoder 150, where the SSD head 160 is overlaid on the trained audio encoder 150.

[0054] At operation 408, the method 400 includes providing a multi-tasking neural network model 130 for execution on the computing device 102, 111 to perform both SID and SSD tasks in parallel on the input audio data 120. The multi-tasking neural network model 130 includes a trained audio encoder 150, a trained SID head 154 overlaid on the trained audio encoder 150, and a trained SSD head 160 overlaid on the trained audio encoder 150.

[0055] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0056] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems and methods described herein. Computing device 500 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0057] Computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 that connects to memory 520 (i.e., memory hardware) and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or exist in other manners as needed. The processor (i.e., data processing hardware) 510 can process instructions stored in the memory hardware (i.e., memory 520) for execution within computing device 500, including instructions stored on memory 520 or storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to the high-speed interface 540. The data processing hardware 510 and memory hardware 520 may include the data processing hardware 113 and memory hardware 115 of the remote system, the data processing hardware 103 and memory hardware 115 of the AED 102, or a combination thereof. In other implementations, multiple processors and / or multiple buses may be used, as well as multiple memories and types of memory, as appropriate. Also, multiple computing devices 500 may be connected, with each device performing some of the required operations (e.g., as a bank of servers, a group of blade servers, or a multi-processor system).

[0058] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0059] The storage device 530 can provide mass storage for the computing device 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be an array of devices, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0060] The high-speed controller 540 manages the more bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages the less bandwidth-intensive operations. Such an allocation of duties is merely an example. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router, for example, via a network adapter.

[0061] The computing device 500 can be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 500a, or multiple times within a cluster of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0062] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0063] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0064] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for performing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0065] To interact with a user, aspects of the invention can be implemented in a computer having a display device, such as a CRT or LCD monitor, or a touch screen, for displaying information to the user, and optionally a keyboard and a pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0066] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (400), when executed by data processing hardware (510), causing the data processing hardware to: obtaining a speaker identification (SID) model (151) trained to predict speaker embeddings (155) from utterances spoken by different speakers, the SID model (151) including a trained audio encoder (150) and a trained SID head (154); receiving a plurality of synthetic speech detection (SSD) training utterances (330), the plurality of SSD training utterances (330) including a set of human spoken speech samples and a set of synthetic speech samples (330b); training an SSD head (160) on the SSD training utterances (330) using the trained audio encoder (150) to learn to detect the presence of synthetic speech in an audio encoding (152) encoded by the trained audio encoder (150), where the SSD head (160) is overlaid on the trained audio encoder (150); providing a multi-tasking neural network model (130) for performing both SID and SSD tasks in parallel on input audio data (120) for execution on a computing device (102), the multi-tasking neural network model (130) including the trained audio encoder (150), the trained SID head (154) overlaid on the trained audio encoder (150), and the trained SSD head (160) overlaid on the trained audio encoder (150); A computer-implemented method (400) for causing an operation to be performed, comprising:

2. 2. The method of claim 1, wherein the SID model is trained on a plurality of SID training data sets, each SID training data set including a plurality of SID training utterances spoken by M different speakers.

3. 3. The method of claim 2, wherein one or more SID training utterances spoken by a particular speaker are paired with a corresponding target speaker embedding (324) to train the audio encoder (150) and the SID head (154) to learn how to predict a corresponding speaker embedding (155) for the particular speaker.

4. 4. The method (400) of claim 2 or 3, wherein the plurality of SID training utterances of one or more of the SID training sets (320) includes N utterances (320) spoken per speaker among the M different speakers.

5. The method (400) of any one of claims 2 to 4, wherein the plurality of SID training utterances of each SID training data set comprises human-generated, non-synthetic speech.

6. The method (400) of any one of claims 1 to 5, wherein the number of speech samples in the set of human spoken speech samples is greater than the number of speech samples in the set of synthetic speech samples (330b).

7. The multitask neural network model (130) comprises: receiving, as an input to the trained audio encoder (150), the input audio data (120), the input audio data (120) characterizing speech captured by an array of one or more microphones (106); generating, as output from the trained audio encoder (150), a shared audio encoding (152) that encodes the input audio data (120); providing the shared audio encoding (152) generated as output from the trained audio encoder (150) as input to the trained SID head (154) and the SSD head (160); generating, as output from the trained SID head (154), an evaluation speaker embedding (155) representative of vocal characteristics of the utterance characterized by the input audio data (120); generating, as output from the trained SSD head (160), an SSD classification score (162) indicative of whether the utterance characterized by the input audio data (120) contains authentic human-generated speech or synthetic speech; The method (400) of any one of claims 1 to 6, configured to perform the SID and SSD tasks in parallel on the input audio data (120) by

8. The multi-task neural network model (130): when the evaluation speaker embedding (155) matches a reference speaker embedding (155) associated with a registered user (10) who is authorized to access the resource; and when the SSD classification scores (162) produced as output from the trained SSD head (160) indicate that the utterance characterized by the input audio data (120) contains authentic human-generated speech; 8. The method of claim 7, configured to authorize access to one or more of the resources.

9. The multi-task neural network model (130): the evaluation speaker embedding (155) does not match any reference speaker embedding (155) associated with a registered user (10) that is authorized to access the resource; or the SSD classification scores (162) generated as output from the trained SSD head (160) indicate that the utterance characterized by the input audio data (120) includes synthetic speech; The method (400) according to claim 7 or 8, further comprising restricting access to one or more of said resources when at least one of:

10. The method (400) of any one of claims 1 to 9, wherein parameters / weights of the trained audio encoder (150) are frozen while training the SSD head (160) with the SSD training utterances (330).

11. The method (400) of any one of claims 1 to 10, wherein the audio encoder (150) comprises a multi-head attention model architecture.

12. The method (400) of claim 11 , wherein the multi-head attention model architecture incorporates a stack of conformer blocks.

13. The method (400) of any one of claims 1 to 12, wherein the SSD head (160) includes one or more fully bonded layers.

14. A system (100), comprising: Data processing hardware (510); memory hardware (520) in communication with the data processing hardware (510), the memory hardware (520) storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to: obtaining a speaker identification (SID) model trained to predict speaker embeddings (155) from utterances spoken by different speakers, the SID model (151) including a trained audio encoder (150) and a trained SID head (154); receiving a plurality of synthetic speech detection (SSD) training utterances, the plurality of SSD training utterances (330) including a set of human spoken speech samples and a set of synthetic speech samples (330b); training an SSD head (160) on the SSD training utterances (330) using the trained audio encoder (150) to learn to detect the presence of synthetic speech in an audio encoding (152) encoded by the trained audio encoder (150), where the SSD head (160) is overlaid on the trained audio encoder (150); providing a multi-tasking neural network model (130) for performing both SID and SSD tasks in parallel on input audio data (120) for execution on a computing device, the multi-tasking neural network model including the trained audio encoder (150), the trained SID head (154) overlaid on the trained audio encoder (150), and the trained SSD head (160) overlaid on the trained audio encoder (150); and memory hardware (520) for performing operations including: A system (100).

15. 15. The system of claim 14, wherein the SID model is trained with a plurality of SID training data sets, each SID training data set including a plurality of SID training utterances spoken by M different speakers.

16. 16. The system (100) of claim 15, wherein one or more of the SID training utterances spoken by a particular speaker are paired with a corresponding target speaker embedding (324) to train the audio encoder (150) and the SID head (154) to learn how to predict a speaker embedding (155) corresponding to the particular speaker.

17. 17. The system (100) of claim 15 or 16, wherein the plurality of SID training utterances of one or more of the SID training sets (320) includes N utterances (320) spoken per speaker among the M different speakers.

18. The system (100) of any one of claims 15 to 17, wherein the plurality of SID training utterances of each SID training data set comprises human-generated, non-synthetic speech.

19. The system (100) of any one of claims 14 to 18, wherein the number of speech samples in the set of human spoken speech samples is greater than the number of speech samples in the set of synthetic speech samples (330b).

20. The multitask neural network model (130) comprises: receiving, as an input to the trained audio encoder (150), the input audio data (120), the input audio data (120) characterizing speech captured by an array of one or more microphones (106); generating, as output from the trained audio encoder (150), a shared audio encoding (152) that encodes the input audio data (120); providing the shared audio encoding (152) generated as output from the trained audio encoder (150) as input to the trained SID head (154) and the SSD head (160); generating, as output from the trained SID head (154), an evaluation speaker embedding (155) representative of vocal characteristics of the utterance characterized by the input audio data (120); generating, as output from the trained SSD head (160), an SSD classification score (162) indicative of whether the utterance characterized by the input audio data (120) contains authentic human-generated speech or synthetic speech; The system (100) of any one of claims 14 to 19, configured to perform the SID and SSD tasks in parallel on the input audio data (120) by

21. The multi-task neural network model (130): when the evaluation speaker embedding (155) matches a reference speaker embedding (155) associated with a registered user (10) who is authorized to access the resource; and when the SSD classification scores (162) produced as output from the trained SSD head (160) indicate that the utterance characterized by the input audio data (120) contains authentic human-generated speech; The system (100) of claim 20 configured to authorize access to one or more of the resources.

22. The multi-task neural network model (130): the evaluation speaker embedding (155) does not match any reference speaker embedding (155) associated with a registered user (10) that is authorized to access the resource; or the SSD classification scores (162) generated as output from the trained SSD head (160) indicate that the utterance characterized by the input audio data (120) includes synthetic speech; The system (100) according to claim 20 or 21, configured to restrict access to one or more of said resources at least one of the following times:

23. The system (100) of any one of claims 14 to 22, wherein parameters / weights of the trained audio encoder (150) are frozen while training the SSD head (160) with the SSD training utterances (330).

24. The system (100) of any one of claims 14 to 22, wherein the audio encoder (150) comprises a multi-head attention model architecture.

25. 25. The system (100) of claim 24, wherein the multi-head attention model architecture incorporates a stack of conformer blocks.

26. The system (100) of any one of claims 14 to 25, wherein the SSD head (160) includes one or more fully bonded layers.

Images